Skip to main content

Lessons from Memory

Started debugging an issue where Linux started calling OOM reaper despite tons of memory is used as Linux cached pages. My assumption was if there is a memory pressure, cache should shrink and leave way for the application to use. This is the documented and expected behavior. OOM reaper is called when few number of times page allocation has failed consequently. If for example mysql wants to grow its buffer and it asks for a page allocation and if the page allocation fails repeatedly, kernel invokes oom reaper. OOM reaper won't move out pages, it sleeps for some time and sees if kswapd or a program has freed up caches/application pages. If not it will start doing the dirty job of killing applications and freeing up memory. In our mysql setup, mysql is the application using most of the Used Memory, so no other application can free up memory for mysql to use. Cached pages are stored as 2 lists in Linux kernel viz active and inactive.
More details here
https://www.kernel.org/doc/gorman/html/understand/understand013.html

Linux tries to move frames from active to inactive list if the cached pages are 2/3 full or the memory used by cached pages have to be shrunk due to low mem threshold. And the inactive cached pages are flushed to disk. So in this OOM incident, it clearly says page cache didn't shrink as fast as needed. There is a open question on this, not sure which would be the right forum to ask this question

https://www.reddit.com/r/kernel/comments/agnt4q/query_on_linux_page_cache/
Now adding 3 to vm.drop_caches will flush page cache in its entirety. It can be done before our peak growth but we wanted to know whats the impact due to that. After reading mysql documentation on innodb buffer pools, we came to know innodb implements its own page cache where it stores the frequently used rows in buffers whose max limit is configured by us as 80% of physical memory. Again when the used memory of buffer reaches 2/3 of its size, mysql will start evicting, writes will reach files when dirty threshold is reached or log files checkpointing have to be done. Since innodb have its own cache and eviction policy of the filebacked databases, maintaining those database files in linux page cache is definitely a duplicity and it is going to skew the least recently used metrics of linux page cache. As a result we disabled the caching of innodb pages by asking mysql to use O_DIRECT to open database files

https://dba.stackexchange.com/questions/226142/innodb-cache-duplication-due-to-os-page-cache-and-buffer-pool

I'm a firm believer that adding swap memory will affect performance and its better to kill the program and start again instead of letting it swap. We started looking at our innodb buffer pool, and our query pattern, it looked our data in buffer pool is not evicted though its not used as the buffer has still place to grow. Adding swap will swap the least used application memory based on vm.swappiness value. 

https://unix.stackexchange.com/questions/32333/what-does-the-vm-swappiness-parameter-really-control

As per the mathematics above swappiness dictate linearly how early kswapd should swap out application pages. If swappiness value is low, only cached file pages will be evicted to disk and if swappiness is very high(close to 100), applications will be swapped before cached pages. Based on our earlier understanding of the problem, we came to the conclusion
1)Earlier there was a increased memory pressure because of cache duplicity and its no longer there due to O_DIRECT
2)There are still some data in page cache which are binlogs, transaction log files.

https://github.com/tobert/pcstat
This is a nice tool which helps us understand what is cached.

When innodb buffer is completely grown, our usage is 51 GB for innodb buffer pool plus overheads(we configured 48GB as innodb buffer pool size). Mysql totally uses 53-54GB of memory where apart from innodb buffer pool memory is used for each connections, sort/join buffer, performance schema buffer and inmemory tables, cached stored procedures(open table cache). So if you configure xGB for innodb buffer pool assume mysql will use 10%-15% over xGB. Our box has a total 61GB. Remaining 6-8GB is used by page cache. And bin logs are cached almost 90%. If there is a flood of writes and bin logs are also updated and mysql requests new pages, there can be a pressure in getting memory from page cache. During that time since we see we have allocated more memory to innodb buffer than needed, using swap will relieve some pressure on memory and move out mysql pages which are not frequently used. If mysql has all its pages as active then swap can affect performance tremendously. We have seen swap used upto 2GB when such cache pressure happens during peak traffic. So swap can be used as an option if application is gradually building memory and is conservative in flushing old data, then OS can do that for you on the event of high memory pressure.

Comments

Post a Comment

Popular posts from this blog

Walking down the Memory Lane!!!

This post is going to be an account of  few trouble-shootings I did recently to combat various I/O sluggishness.
Slow system during problems with backup
We have a NFS mount where we push backups of our database daily. Due to some update to the NFS infra, we started seeing throughput of NFS server drastically affected. During this time we saw general sluggishness in the system during backups. Even ssh logins appeared slower. Some boxes had to be rebooted due to this sluggishness as they were too slow to operate on them. First question we wanted to answer, does NFS keep writing if the server is slow? The slow server applied back pressure by sending small advertised window(TCP) to clients. So clients can't push huge writes if server is affected. Client writes to its page cache. The data from page cache is pushed to server when there is a memory pressure or file close is called. If server is slow, client can easily reach upto dirty_background_ratio set for page cache in sysctl. This di…

How we have systematically improved the roads our packets travel to help data imports and exports flourish

This blog post is an account of how we have toiled over the years to improve the throughput of our interDC tunnels. I joined this company around 2012. We were scaling aggressively then. We quickly expanded to 4 DCs with a mixture of AWS and colocation. Our primary DC is connected to all these new DCs via IPSEC tunnels established from SRX. The SRX model we had, had an IPSEC throughput of 350Mbps. Around December 2015 we saturated the SRX. Buying SRX was an option on the table. Buying one with 2Gbps throughput would have cut the story short. The tech team didn't see it happening.

I don't have an answer to the question, "Is it worth spending time in solving a problem if a solution is already available out of box?" This project helped us in improving our critical thinking and in experiencing the theoretical network fundamentals on live traffic, but also caused us quite a bit of fatigue due to management overhead. Cutting short the philosophy, lets jump to the story.

De…