Skip to main content

Lessons from Memory

Started debugging an issue where Linux started calling OOM reaper despite tons of memory is used as Linux cached pages. My assumption was if there is a memory pressure, cache should shrink and leave way for the application to use. This is the documented and expected behavior. OOM reaper is called when few number of times page allocation has failed consequently. If for example mysql wants to grow its buffer and it asks for a page allocation and if the page allocation fails repeatedly, kernel invokes oom reaper. OOM reaper won't move out pages, it sleeps for some time and sees if kswapd or a program has freed up caches/application pages. If not it will start doing the dirty job of killing applications and freeing up memory. In our mysql setup, mysql is the application using most of the Used Memory, so no other application can free up memory for mysql to use. Cached pages are stored as 2 lists in Linux kernel viz active and inactive.
More details here
https://www.kernel.org/doc/gorman/html/understand/understand013.html

Linux tries to move frames from active to inactive list if the cached pages are 2/3 full or the memory used by cached pages have to be shrunk due to low mem threshold. And the inactive cached pages are flushed to disk. So in this OOM incident, it clearly says page cache didn't shrink as fast as needed. There is a open question on this, not sure which would be the right forum to ask this question

https://www.reddit.com/r/kernel/comments/agnt4q/query_on_linux_page_cache/
Now adding 3 to vm.drop_caches will flush page cache in its entirety. It can be done before our peak growth but we wanted to know whats the impact due to that. After reading mysql documentation on innodb buffer pools, we came to know innodb implements its own page cache where it stores the frequently used rows in buffers whose max limit is configured by us as 80% of physical memory. Again when the used memory of buffer reaches 2/3 of its size, mysql will start evicting, writes will reach files when dirty threshold is reached or log files checkpointing have to be done. Since innodb have its own cache and eviction policy of the filebacked databases, maintaining those database files in linux page cache is definitely a duplicity and it is going to skew the least recently used metrics of linux page cache. As a result we disabled the caching of innodb pages by asking mysql to use O_DIRECT to open database files

https://dba.stackexchange.com/questions/226142/innodb-cache-duplication-due-to-os-page-cache-and-buffer-pool

I'm a firm believer that adding swap memory will affect performance and its better to kill the program and start again instead of letting it swap. We started looking at our innodb buffer pool, and our query pattern, it looked our data in buffer pool is not evicted though its not used as the buffer has still place to grow. Adding swap will swap the least used application memory based on vm.swappiness value. 

https://unix.stackexchange.com/questions/32333/what-does-the-vm-swappiness-parameter-really-control

As per the mathematics above swappiness dictate linearly how early kswapd should swap out application pages. If swappiness value is low, only cached file pages will be evicted to disk and if swappiness is very high(close to 100), applications will be swapped before cached pages. Based on our earlier understanding of the problem, we came to the conclusion
1)Earlier there was a increased memory pressure because of cache duplicity and its no longer there due to O_DIRECT
2)There are still some data in page cache which are binlogs, transaction log files.

https://github.com/tobert/pcstat
This is a nice tool which helps us understand what is cached.

When innodb buffer is completely grown, our usage is 51 GB for innodb buffer pool plus overheads(we configured 48GB as innodb buffer pool size). Mysql totally uses 53-54GB of memory where apart from innodb buffer pool memory is used for each connections, sort/join buffer, performance schema buffer and inmemory tables, cached stored procedures(open table cache). So if you configure xGB for innodb buffer pool assume mysql will use 10%-15% over xGB. Our box has a total 61GB. Remaining 6-8GB is used by page cache. And bin logs are cached almost 90%. If there is a flood of writes and bin logs are also updated and mysql requests new pages, there can be a pressure in getting memory from page cache. During that time since we see we have allocated more memory to innodb buffer than needed, using swap will relieve some pressure on memory and move out mysql pages which are not frequently used. If mysql has all its pages as active then swap can affect performance tremendously. We have seen swap used upto 2GB when such cache pressure happens during peak traffic. So swap can be used as an option if application is gradually building memory and is conservative in flushing old data, then OS can do that for you on the event of high memory pressure.

Comments

Post a Comment

Popular posts from this blog

How we have systematically improved the roads our packets travel to help data imports and exports flourish

This blog post is an account of how we have toiled over the years to improve the throughput of our interDC tunnels. I joined this company around 2012. We were scaling aggressively then. We quickly expanded to 4 DCs with a mixture of AWS and colocation. Our primary DC is connected to all these new DCs via IPSEC tunnels established from SRX. The SRX model we had, had an IPSEC throughput of 350Mbps. Around December 2015 we saturated the SRX. Buying SRX was an option on the table. Buying one with 2Gbps throughput would have cut the story short. The tech team didn't see it happening. I don't have an answer to the question, "Is it worth spending time in solving a problem if a solution is already available out of box?" This project helped us in improving our critical thinking and in experiencing the theoretical network fundamentals on live traffic, but also caused us quite a bit of fatigue due to management overhead. Cutting short the philosophy, lets jump to the story.

LXC and Host Crashes

 We had set up a bunch of lxc containers on two servers each with 16 core CPUs and 64 GB RAM(for reliability and loadbalancing). Both the servers are on same vlan. The servers need to have atleast one of their network interface in promiscuous mode so that it forwards all packets on vlan to the bridge( http://blogs.eskratch.com/2012/10/create-your-own-vms-i.html ) which takes care of the routing to containers. If the packets are not addressed to the containers, the bridge drops the packet. Having this setup, we moved all our platform maintenance services to these containers. They are fault tolerant as we used two host machines where each host machine has a replica of the containers on the other. The probability to crash for both the servers at the same time due to some hardware/software failure is less. But to my surprise both the servers are crashing exactly the same time with a mean life time 20 days. We had to wake up late nights(early mornings) to fix stuffs that gone down The

The server, me and the conversation

We were moving a project from AWS to our co-located DC. We have setup KVMs scheduled by Cloudstack for each of the component in the architecture. The KVMs used local storage. The VMs are provisioned with more than required resources because we have the opinion that in our DC scaling during peak load and then downscaling doesn't offer much benefits financially as we are anyways paying for the hardware in advance and its also powered on. Its going to be idle if not used. Now we found something interesting our latency in co-located DC was 2 times more than in AWS. The time for first byte at our load balancer in aws was 60ms average and at our DC was 112ms. We started our debugging mission, Mission Conquer-AWS. All the servers are newer Dell hardwares. So the initially intuition was virtualisation is causing the issue. Conversation with the Hypervisor We started with CPU optimisation, we started using the host-passthrough mode of CPU in libvirt so VMs dont see QEMU emulated CPUs,