We were facing a weird issue on our almost stable dns infrastructure. We use PDNS servers and our custom backend to serve DNS requests. So the PDNS server pipes the request to custom backend, the backend reads from the pipe and puts the result back in the pipe which is served back to the client by the server. This intro is not at all useful to the post. So please ignore it.
The problem started since Monday where randomly our PDNS backend started using 100% CPU and the PDNS server started crashing. The backend was written in C++ and the debugging started. The backend had no debug mode to start with (!). So we attached all backend processes to strace. Strace showed a particularly crafted DNS request put the backend in infinite loop. We couldn't get the whole request in the strace as it was bigger than default 32 bytes. We started strace with -s 5000 to capture 5000 bytes. Now the dns request is found. The domain name is 312 bytes long. A full domain should not exceed 253 characters ideally. Dig wont allow one to use 312 bytes long domain name. So this dns packet could be crafted.
As we got the input, we started debugging the code with gdb. What we found wierd was cin stops reading from pipe suddenly. The line cin was not waiting for input. We can imitate it by writing a code like
int main(){
int n;
while(true){
cin>>n;
count<<"got input";
}
}
At any point when cin is waiting for 'n', if we enter a string cin sets an error flag and stops reading from the input buffer. So "got input" will be printed infinitely from the point n got a string instead of integer.
cin.clear() and cin.ignore() will come in rescue during this situation. This stack overflow link should give more details on this behaviour of C++.
The problem started since Monday where randomly our PDNS backend started using 100% CPU and the PDNS server started crashing. The backend was written in C++ and the debugging started. The backend had no debug mode to start with (!). So we attached all backend processes to strace. Strace showed a particularly crafted DNS request put the backend in infinite loop. We couldn't get the whole request in the strace as it was bigger than default 32 bytes. We started strace with -s 5000 to capture 5000 bytes. Now the dns request is found. The domain name is 312 bytes long. A full domain should not exceed 253 characters ideally. Dig wont allow one to use 312 bytes long domain name. So this dns packet could be crafted.
As we got the input, we started debugging the code with gdb. What we found wierd was cin stops reading from pipe suddenly. The line cin was not waiting for input. We can imitate it by writing a code like
int main(){
int n;
while(true){
cin>>n;
count<<"got input";
}
}
At any point when cin is waiting for 'n', if we enter a string cin sets an error flag and stops reading from the input buffer. So "got input" will be printed infinitely from the point n got a string instead of integer.
cin.clear() and cin.ignore() will come in rescue during this situation. This stack overflow link should give more details on this behaviour of C++.
Great dude.
ReplyDelete