Monday, April 15, 2013

Fast DLI Downloader Tool to download PDF Books from Digital Library Of India

Digital Library Of India [http://www.dli.ernet.in/] hosts millions of copyright freed scanned books in English and Indian Languages for public access. Unfortunately this website does not provide any mechanism to download the entire e-book in any format. DLI-downloader provides a smooth and hassle free interface to all DLI servers which allows you to search and downloads the eBooks right from the Tool.

Current status of Book Collection at DLI - http://www.dli.gov.in/cgi-bin/status.cgi

Download this Tool from here - http://dli.shunyafoundation.com/

Video Demonstration of the Tool



 5 minutes Video Introduction for dli-downloader Tool (Play HD)

Key Features of DLI Downloader

  1. Search DLI books at lightening fast speed. Option to find the similar books.
  2. Download Books in PDF Format right within the DLI Downloader tool with a single click.
  3. Queue up your download items in the tool and let it download books silently for you.
  4. Load balancing from multiple dli-servers for better speed for downloads.Tool will pick up the best speed available DLI server for download.
  5. Application can minimize to system tray so as not to interfere with your other tasks & reducing the memory footprint. It saves unfinished jobs upon exit.
  6. Configure the download speed threshold of the application over a metered network so that the tool does not eat up all the available bandwidth of the network.
  7. Automatic error handling for the corrupt file downloads due to unknown errors at the time of PDF conversion.
  8. Internet Connectivity Issues - not a problem with this tool. Downloads will pause when your internet connection goes down and resume after internet is back. what else you want ?
  9. Works well on x64 & x86 on Windows 7, Windows 8, Mac And Ubuntu 12.10.
  10. Above all, everything is for Free !!

How it Works ?

Searching a Book in Digital Library of India is very easy through this tool, just type in the keyword in search panel. Existing downloads appears in green color, that you can directly open. For other titles, just right click and choose download.

Searching for the Desired Book




Downloading the Book in PDF format

Sometimes we want to configure settings for the dli downloader, the Settings Tabs is right for this work. Don't change the settings if you do not understand it.

Configuring the Settings


System Requirements

  1. Java 8 runtime environment (http://java.com/en/download/index.jsp)
  2. Windows 7, Windows 8, Mac, Ubuntu 12.10 x64 & x86
  3. Latest DLI Downloader from http://dli.shunyafoundation.com/

How To Run ?

  1. Run the dli-downloader jar by double clicking it on windows. Unix and Mac users please do run the following command
    java -jar dli-downloader-5.9-jar-with-dependencies
  2. Create Local Index for Books catalogue for a language of you choice, this step is must for enabling the local search (otherwise you need to manually enter barcode into downloads Tab)
  3. Search the desired book in Search Tab and right click to download the book.
  4. Your download will start or queue up.
  5. Once you are done with all the downloads, then find the DLI icon in the system tray and right click on it, then choose exit to quit the application.
  6. That's it, enjoy !!!

Configuration Settings Explained

  • numberParallelJobs - number of parallel downloads at any given point in time.
  • readTimeOutMs - this is socket read time out value before throwing an timeout exception in milli seconds.
  • rootDirectory - directory where the fresh downloads will be saved.
  • speedLimitKBps - maximum speed limit that the application will use for all its downloads.
  • deleteTifIfSuccessful - if set to true then the TIFF files will be deleted if PDF conversion and tiff download goes successful.
  • maxRetryCount- If connection to server is very slow then the connection timeout exception can occur. In case the download fails, then this property will decide how many times to retry the failed download.
  • maxConsecutiveFailure - maximum number of consecutive IOExceptions from the server before the tool stops downloading the given barcode.
  • downloadDirectories- comma or semicolon separated list of directories which can contain the already downloaded PDF files. The pdf file must contain the barcode in its name. This directory will be scanned by the tool at the time of startup for caching the existing downloads and a warning prompt will appear to user if he tries to download the same barcode again.
  • quality - TIFF to PDF conversion quality. Maximum quality is A2, minimum is A7
  • lookAndFeelNumber - 0 is SystemLookAndFeel, 1 is NimbusLookAndFeel, 2 is WindowsLookAndFeel
  • createBarcodePage - setting it to true will create an initial page displaying the metadata of the barcode.

Java Source Key Features 

The Java Implementation of this tool embeds various design patterns & Algorithms like,
  1. Token Bucket Algorithm for managing the download speed
  2. Observer Design pattern which enables Swings GUI to be observer of the running download task
  3. Concurrency Utilities like Executor Framework has been utilized for the parallel processing and maximized throughput.

Download the latest version from GitHub

Download the latest version from the below location -
http://www.shunyafoundation.com/dli-downloader-v6.0.2/dp/C755E13VGS


Donation (As little as Rs 100 matters)

If anyone feel like donating money for this work, feel free to do that. Even a small amount will help us support the cause (Distributing Vedic Literature, Tree Plantation & Gow Palan, etc)

Where do we spend the funds ?
http://dli-downloader.blogspot.in/2013/12/charity-for-prosperity-where-fund-goes.html

Bank Deposit by NEFT

MUNISH CHANDEL
a/c 5277618224
IFSC : CITI0000002
Citi Bank N.A. Delhi

Don't forget to send me the details on email : cancerian0684@gmail.com


Feedback

For any feedback or suggestions, write back to me cancerian0684@gmail.com or leave a comment on this webpage.



75 comments:

  1. i gone through your website its really good information, we are also using much similar PRODUCTS, You can visit us.

    BARCODE INIDA



    .

    ReplyDelete
  2. Munish,

    Your software has fulfilled my long cherished wish. I have been using IDM till now for downloading from DLI. I downloaded and tried your DLI downloader just now. It is pretty fast compared to previously available tools. Another good feature is that one can add any number of books for download as long as internet connection is fast enough.

    Only drawback of this tool is that page size is changed to A4 size. But, this problem is negligible compared to the good features of the software.

    Thank you very much for developing this wonderful software.

    ReplyDelete
  3. Hi Shankara, thanks for using this software and providing the right feedback. I have changed to TIFF to PDF quality to A2 (that is the scanning quality of DLI scanner, I believe), Please use Version 2.6 to avail this change
    http://code.google.com/p/dli-downloader/downloads/list

    ReplyDelete
  4. dear manish,

    can i use this downloader in MAc machine? Please iform me. shrivara@gmail.com

    ReplyDelete
    Replies
    1. after installing Java 7 on Mac, you can run the following command on the command prompt

      java -jar DLIDownloader-2.9-jar-with-dependencies.jar

      that should work

      Delete
    2. And please do let me know if it works on Mac or not. I am keen to know that !!

      Delete
    3. Hi SamanA,
      We have now tested it on Mac as well and it works just fine. So just go ahead and enjoy the software on your macbook :)
      P.S. : Due to security restrictions, Mac wont open the file the first time try to by double clicking it. You will have to right click and select Open With option. Next time onwards, it will open with double click as well.

      Delete
  5. This comment has been removed by a blog administrator.

    ReplyDelete
  6. Munish,

    You have really done a great job ! The multiple download feature is the most desireable one for all DLI lovers. Also, you have changed the page size from A4 to A2. I was using IDM previously, but as you have added the multiple download feature, I am very happy to use your downloader. Thank you very very much !

    ReplyDelete
  7. Hi Shankara, I have rolled out new version of software accommodating few new changes -
    1.) Remove all option on the GUI
    2.) Configurable PDF Quality (A2 to A7) in dli-settings.xml, automatic detection is tough to implement because few barcodes have different sized TIFFS for different pages. A2 is the maximum quality, A7 is minimum.

    ReplyDelete
  8. Hi Munish

    I have a requirement. I can download all the tiff files for a given barcode number to a folder. Now converting that tiff files to Pdf can I use your classes to do it. If this is possible can you please help me to know how to do that in a simple java file. That would be a much helpful for me.

    Thanks
    C.R.Bala
    Bangalore

    ReplyDelete
    Replies
    1. There are two ways to do it -
      1.)
      a) Place all the TIFF files in a folder and name this folder as the barcode i.e. 5990010100576
      b) Now start this tool and configure the suitable download location i.e. E:/downloads
      c) then cut and paste your barcode folder into this location.
      d) now enter this barcode into tool's download list, tool will scan the metadata from web and then try to download the TIFF files into barcode folder. But seeing the already existing tiff files, it will not download them again. Tool will just reuse your tiff files and convert them to PDF

      I have done this many times and should definitely work for you.

      2) You can grab the source code for TiffToPDF.java from
      https://code.google.com/p/dli-downloader/source/checkout and try importing project into inellij/etc
      This file does the conversion using iText as the underlying library.

      3) Just check out how iText libray works, its very simple to convert tiff to pdf using iText

      Delete
  9. How about Linux users sir, whether the same tool works? Or do we have a Linux version too? Please clarify and provide link for the Linux version as needed. Thanks.

    ReplyDelete
    Replies
    1. I have tested this tool in Ubuntu 12.10 x64 and it works perfectly fine (though i tested few versions of it only). The only thing that you need is Java 7 installed on ubuntu. Download the jar from https://code.google.com/p/dli-downloader/downloads/list
      and then run the following command

      java -jar DLIDownloader-2.9-jar-with-dependencies.jar

      Delete
  10. Thanks for the tool Munish.
    I had trouble downloading initially it used to fetch book details and start the download and change the status to Failed immediately.
    Then i realized i am not running latest version of Java. And after updating the java it is working absolutely fine.

    ReplyDelete
    Replies
    1. great, In case you are a developer and understand Java, then you can always run the tool from command prompt using following command

      java -jar DLIDownloader-2.9-jar-with-dependencies.jar

      This will print every exception details on the command prompt making debugging/ troubleshooting easy.

      Delete
  11. Thanks for creating a nice tool for downloading books from DLI.
    Although, this tool worked well for me, I'm now seeing this in terminal :
    D:\DLI>java -jar DLIDownloader-3.0-jar-with-dependencies.jar
    [AWT-EventQueue-0] INFO org.shunya.dli.Utils - Loading File : dli-settings.xml
    [AWT-EventQueue-0] INFO org.shunya.dli.Utils - Loading File : dli-jobs.xml
    Could not load XML file : dli-jobs.xml
    javax.xml.bind.UnmarshalException
    - with linked exception:
    [java.io.FileNotFoundException: D:\DLI\dli-jobs.xml (The system cannot find the
    file specified)]
    I'm on Windows 8. I'm using DLI downloader 3.0.
    When I enter barcode and press 'Add', I can't see anything added. I did it multiple times. I've tried shutting application down and then running again. I've deleted old versions and xmls. Nothing seems to work.
    Although, I'm now updating Java in hope to get it working.
    Any suggestions ?

    ReplyDelete
    Replies
    1. Hi Dear,
      The exception that you are getting on console is normal (UnmarshalException) and will happen just for the first time when their is no job file created. I wonder why it is not picking up the job, because i have tested version 3.0 on windows 8 itself and it works just great. Even Java version should not be a problem as long as it is Java 7. Can you provide me some more information on my email : cancerian0684@gmail.com

      1) java -version
      2) does DLI 2.9 version work on your system ? https://dli-downloader.googlecode.com/files/DLIDownloader-2.9-jar-with-dependencies.jar
      3) Which barcode are you trying to add ?
      4) does the screen remain empty even after adding barcode ?
      5) can you put the barcode into a text file and put the filepath of that text file into tool.
      6) Can you send the complete logs from console after you try adding a barcode into the tool ?

      Delete
    2. Please do update if it worked well for you or not ?

      Delete
    3. 1. java-version : 7.21
      2. It works with same problems persisting.
      3. Anonymous barocdes were used.
      4. Yes.
      5. Putting barcodes in job.xml file works for me.
      6. The previously sent log was complete.

      Delete
    4. It seems like there is some configuration on your system which is not respected by the tool and it fails. I will try to create a new build for getting more clues for the same. Will update you once ready.. that build will just help me finding the real problem that is happening.

      Delete
    5. Thanks for looking after the problem.
      Although, I can assure you that I've this software working fine and without changing anything in my System Settings or Java Settings, it started behaving like that.

      Delete
  12. Thank you very much for this great and useful tool, Munish!

    One question: when the PDF gets created, it produses a barcode page as its first page. Is there an option to leave this out?

    Thanks again for this great job!

    ReplyDelete
    Replies
    1. Hey you can download version 3.1 from
      https://code.google.com/p/dli-downloader/downloads/list

      Right click on the system tray icon for DLI and choose settings.

      Choose this property and set it to false (by double clicking on it)
      createBarcodePage false

      this does not require a tool restart, but make sure you shutdown the tool using shutdown option otherwise it will not save the settings upon exit.

      This should solve your problem.

      Delete
    2. Thank you very much for your concern. It works perfectly!
      Best wishes.

      Delete
  13. When I shutdown the downloader from notification area, it just saves running barcodes in job.xml. queued barcodes are not saved.

    Another problem I'm facing is: if internet connection drops and resumes, you will have many corrupt .tiff files. DLI downloader considers them complete and skips to download left part. It will be better if it could match size of tiff to that on server.
    When such corrupt files are situated in tiff directory, DLI downloader could not make pdf - is also observed.

    ReplyDelete
    Replies
    1. I will look for the reasons why tool does not save the queued items at the time of shutdown.
      Regarding partial download of the TIFF files in intermittent connections, many times it is not possible to get the file size from the server to match against. But in TIFF to PDF conversion when such partially downloaded files fail to convert, tool retry once more to re-download the TIFF file and do the conversion once again. So Ideally even if there are partial downloads, PDF conversion should bridge the gap and re-download the files if there is internet connection available at that time.

      Delete
    2. Thanks for considering issues presented by me.
      When creating PDF from partially downloaded tiff files, the DLI downloader gives me 'FAILED' error.

      Delete
    3. when you get "FAILED" error for partially downloaded file, right click on the task and select "View Logs". Then copy all the contents and email to me cancerian0684@gmail.com

      Without logs, it would be hard for me to comment anything.

      Delete
    4. Whenever something fails, a log file is created in the download directory with name .log
      Please send that to me over the email.

      Delete
    5. With version 3.4 my problem of failed conversion is solved. Partial downloads are completed by software before completing conversion.
      This version also saves my queued tasks well.
      Although, I observed that right clicking notification icon and shutting it doesn't shut it always. A click on that icon brings the software back. This is unexpected.
      One more thing, right clicking running task and stopping it is causing hang.
      Thanks for solving my previous problems.

      Delete
    6. Shutdown takes a lot of time because it tries to gracefully Stop all the running Task, if possible so that no file is left in corrupt state. It is little tricky to handle quicker shutdown but I have given a try to handle it in slightly better way. Can you please check it with version 3.5
      The same change should solve the stopping task hang problem.
      I would say the hang problem is definitely solved, but shutdown problem should be slightly improved.

      Delete
  14. I don't know whether others have noted it or not, but I've seen that in some cases the actual number of pages on server is more than the listed one.
    I found it when using another method of downloading DLI books.
    If possible make sure that DLI Downloader does check that.

    ReplyDelete
    Replies
    1. Earlier I was just checking for the TotalPages, now I have added functionality to check for the extra pages in the end. But I have no such barcode to test, can you help testing the same in version 3.5

      Delete
  15. Munishji,

    Downloaded and tried 4.1 version. It has some wonderful new features, especially the search facility. "Remove all completed" is also a good feature. Thanks a lot for devoting your valuable time for great tool.

    ReplyDelete
  16. Tried to download tamil books, but search is not listing the same. Pls help

    ReplyDelete
    Replies
    1. Please download version 4.2 from http://code.google.com/p/dli-downloader/downloads/list

      Goto Downloads Tab and right click on the panel. Select "Index Language" option and then type Tamil in there. Wait for the indexing to complete and search should work fine after that. Please do let me know if that works for you.

      Delete
    2. Munisji,

      Thank you. Did as per your advice. downloads working fine. thank you once again.

      Delete
    3. Fantastic tool and great response. I needed this reply - was trying to figure out how to download Bangla books.

      Delete
  17. Hi! Great program, but some problem! ^-)))
    While downloading - Bagchi, P.C., ed. Visva-Bharati Annals vol. 5 RELIGION. THEOLOGY - didn't try any other - PDF file created contains pages in total disorder. It's impossible to read the file.
    Is it possible to order pages in a right way?

    ReplyDelete
    Replies
    1. I tried downloading the mentioned book (barcode : 4990010203965) and did not see any problem, Can you send me the pdf file to my email - cancerian0684@gmail.com

      Delete
    2. I have fixed a bug which was causing page ordering problem in UNIX environment, you should checkout version 4.4 for the fix.

      Delete
  18. not able to dwnload kannada books..plz help

    ReplyDelete
  19. Initially it worked very well but after few downloads it became very slow to download the books , no change in internet speed. so how can is resolve this issue?

    ReplyDelete
  20. That could be because of heavy load on the DLI servers, otherwise there is no logic in DLI downloader to downgrade the speed after usage. You can goto Settings Tab of the tool and check the maximum speed for download. If required change the value to some upper limit.

    ReplyDelete
  21. Thanks for your excellent tool. I have a problem though. It seems that the tool does not recognize some of the barcodes, say for example 99999990330890. when I put this number in the search field, no results are returned. It happens with the barcodes starting with 9, I have noticed. With barcodes starting with 4 there is no problem. I have tried both version 4.3 and 4.4. I would be grateful if you manage to take a look at this and tell me what exactly is going wrong. I use Windows 7 Home Premium OS and JAVA Version 7 update 40.

    ReplyDelete
    Replies
    1. 99999990330890 belongs to Bengali collection of books.
      Downloading should work for this. Goto Downloads Tab, enter the barcode there and press enter.
      For enabling searching, you need to go to Downloads Tab and right click on the middle panel, choose and then type in Bengali there. Now wait fore the indexng to complete. Once index job is complete, you can search all Bengali Titles. Do let me know if you face any problem.

      Delete
    2. Hello Munish, many thanks for getting back so soon. I did what you told me a to do and it worked! Generally what I do is that I type or copy paste the barcode in the SEARCH tab first. In my experience it almost always returns the title I am looking for. Then I Right Click on the title (displayed in the search result) and select download. In this case, the search tab is not indexing the the book I am looking for. Following your method however, I have been able to solve my problem, so thanks again.

      Delete
  22. This comment has been removed by the author.

    ReplyDelete
  23. Hi Munish,

    Have you developed this software also for android... I'm using Samsung galaxy s2. I don't have a laptop or a desktop computer.

    It wud be wonderful if you could provide your software for people like me also who have access only to an android device.

    ReplyDelete
  24. Munish,

    i am using samsung chromebook and would like to know how to install and use this downloader on samsung chromebook

    ReplyDelete
  25. Hi Munish

    I have downloded the 4.5 version in my system. and tried to download one book. But the book is not downloading. The reason being in the logs Fetching metadata for Barcode : 1990020083157
    java.net.SocketTimeoutException: connect timed out
    java.net.SocketTimeoutException: connect timed out
    But I can access internet and other sites but when trying to download from this jar file I can see it is connection timed out. Then I opened the DLI site and tried to open the same barcode number. I could view the book properly.

    any help would be appreciated.

    Thanks
    C.R.Bala

    ReplyDelete
    Replies
    1. Hi,
      Are you able to download any other book using this tool ? Is there a proxy settings on your machine (sometimes proxy server is used in office premises)
      Which operating system you are using currently ?

      Delete
    2. Hi Munish

      Can't download any other books using this tool. Yes there is a proxy server used in the office.

      The Operating System is Windows 7. Also when I run this same in my home laptop it is working fine. I found one thing in my home machine. When it tries to download the book barcode that i mentioned earlier. it tries to downloading like this www.new.dli.ernet.in//data_copy/upload/0083/162/PTIFF. but I tried by copying this URL in my Office Laptop and try to access it. It says that Access is denied/system needs to log on. Any workaround to access this URL. How is it possible in the Java program to access this URL. Somekind of workaround should have been done. Please do tell me.

      Thanks
      C.R.Bala

      Delete
    3. Can you please try step 5 in the mentioned link for proxy settings -

      http://www.shunya.co.in/p/1.html

      Delete
    4. HI Munish

      I tried like this D:\>java -client -Dhttp.proxyHost=10.0.0.100 -Dhttp.proxyPort=8800 -jar dli-down
      loader-4.5-jar-with-dependencies.jar
      Error: Unable to access jarfile dli-downloader-4.5-jar-with-dependencies.jar

      It is giving an error like "Error: Unable to access jarfile dli-downloader-4.5-jar-with-dependencies.jar"

      Please help me to resolve this.

      Thanks
      C.R.Bala

      Delete
    5. you should always run this command from directory where this jar is present. So open command prompt, cd and then run the command.

      Otherwise you can try copying the dli jar to d:/ and then run your previous command that you mentioned in the blog.

      Delete
  26. Hi Munish

    I tried by giving the same command in the directory where the jar is present. now a different error is coming. Java.net Socket Connection timed out. please help me how to open this url in the firefox. Because if this URL is accessible through Java socket it should be accessible via firefox. Please help me out how to access this url. "www.new.dli.ernet.in//data_copy/upload/0083/162/PTIFF"

    Thanks
    C.R.Bala

    ReplyDelete
  27. I'm not able to download books for the last 10 days. Is anyone else facing the same problem?

    ReplyDelete
  28. Munish, it downloads the first 4-5 pages and then doesn't progress further. Can you check whether it's working for you or not?

    ReplyDelete
  29. seems to be some problem with the dli servers, they are not allowing fast downloads by clients. Can you try settings the lower value for the speed limit (80 KBPS) in the dli-settings.

    ReplyDelete
    Replies
    1. I reduced the max speed to below 80 kbps but it's still not downloading more than the first few a pages.

      I tried downloading with Internet Download Manager. The curious thing is that a book from dli.ernet.in is being downloaded on IDM normally without any issues and without any speed limiter in effect. I was able to download the whole book from ernet.in in decent time.

      But when I tried to download a book from dli.gov.in, it stopped downloading after the first few pages. Just like it's happening in your programme.

      Does your programme download books from dli.gov.in or dli.ernet.in? Does it make any difference? Btw, are you encountering the same problem?

      Delete
  30. is this available for android mobiles ?
    if yes then how to go with it

    ReplyDelete
  31. Hi Munish,

    It is wonderful tool, you have fulfilled the gap of pdf download from DLI. To fulfil DLI mission it is GOOD tool to spread knowledge easily. I can confirm that GOD will bless you and your team to helping the people.

    Thanks
    Raja

    ReplyDelete
  32. This comment has been removed by the author.

    ReplyDelete
  33. Thanks, Munish for a good software.
    I've been watching two problems:
    1. The download speed is not very high, although I've a good connection. It appears as if the software is downloading one page/2minute, i.e. snail speed.

    2. If I go on adding barcode more than 100(or like that), the software shows them, but when restarted it removes many to limit their number under 100.

    If possible, deal with these problems.

    ReplyDelete
    Replies
    1. Hi Lalit,
      For your first issue regarding the download speed - All DLI servers are heavily loaded due to huge demand for DLI books, we can not do much on the dli-downloader to increase the speed because of limitations on the server side.
      2nd issue that you mentioned is because of the design, I hope you respect this limit because otherwise non-human users can go on downloading the books forever. 100 is quite a reasonable number for download queue.

      Delete
    2. You can definitely try one thing - dli-settings has list of comma separated dliServers which it connects to. If you know that a particular dli server is performing well at the moment, then you should just put that entry (and remove all others servers) into the settings and restart the tool.

      Delete
    3. some good news - try the new version for better speed (this version will dynamically choose best speed dli-server)

      http://dli-downloader.googlecode.com/files/dli-downloader-4.8-jar-with-dependencies.jar

      Delete
    4. Thanks, Munish.
      This is what I was expecting.
      I will report you soon.

      Delete
    5. Thanks, Munish.
      This version is utilizing full capacity of my internet connection. Now, I'm able to get books fast.

      Delete
  34. Fantastic job... i owe you a big time... u saved my lifetime.. sure i'll donate you this month end.. keep working on things which cannot be valued... i started downloading the tamil literature/medical/philosophical rare and very old book published 150 years ago based on tamil scripts which was done 5000 years ago.. thanks once again.. NANRI

    ReplyDelete
  35. Added new version to google code addressing the download speed related issues. Also added video demonstration of the Tool on YouTube.

    ReplyDelete