There was a huge attack on our infrastructure couple of days before. It didnt follow the regular pattern. So did a lot of googling (google verb!) but eventually we were not to able to successfully find the intricacies employed in the attack. During this literature reference, I came across Partial Get feature in HTTP header. This is the feature used by Download Managers like IDM by spawning multiple threads to download a file. The partial Get request specifies the byte ranges it requires to download.
Eg
The request requires first 89bytes of the file. There can be more than one chunk requested in the range header. The webserver responds with a 206 code for partial GET.
I have coded a primitive threaded downloader which spawns 10 threads, downloads 10 different chunks of a big file and unite them as a single file. Committed the initial version at github https://github.com/kalyanceg/downloader/blob/master/curler.java. Please do feel free to checkout the code (I will add documentation soon) and develop UI/more feature over it.
To view a demo, checkout the code and run
Eg
GET / HTTP/1.1
Host: 127.0.0.1
Range: bytes=0-89
The request requires first 89bytes of the file. There can be more than one chunk requested in the range header. The webserver responds with a 206 code for partial GET.
I have coded a primitive threaded downloader which spawns 10 threads, downloads 10 different chunks of a big file and unite them as a single file. Committed the initial version at github https://github.com/kalyanceg/downloader/blob/master/curler.java. Please do feel free to checkout the code (I will add documentation soon) and develop UI/more feature over it.
To view a demo, checkout the code and run
java curler <input url>.Note: No output will be displayed on the prompt. Will add them to improve usability. Once the program finishes, the file would be downloaded in the same directory where the code is.
Did you benchmark this?
ReplyDelete