Using VisualWget to download websites

Last revision February 17, 2011

What is VisualWget?

Wget is an application to download content from websites. It can be setup to download entire websites by running a single command, without requiring any user intervention. It was originally written for Unix and Linux and must be run from a command line, with all configuration and options specified on the command line in standard Linux formats.

VisualWget makes it easy to run Wget on Windows by giving you a visual interface with check boxes and data entry fields. You can setup all your options and then tell the program to run the command. It works on Windows XP and requires the Microsoft .NET Framework version 2.0 (later versions will probably also work). It may work on Windows Vista or 7 but is not verified by the author.

This web page describes suggested options for using VisualWget to download the entire contents of a website, starting from a single URL, and following all links on that page. Links to other websites (domains) are not followed. The directory structure of the original website is duplicated on your local hard drive (in the folder of your selection), and all files from the website, including html pages, images, pdf files, etc., are downloaded into the appropriate directories.

If the website uses a dynamic content management system (such as Wordpress or Drupal), this program will download a static representation of the site. It cannot download the scripts and databases that underlie the site.

Get VisualWget and verify that it runs.

Open the browser of your choice on your Windows XP computer. I used Firefox. Go to the VisualWget application site at

Click on the big "Download Now" button at the top of the page. This will download a zip archive file of about 1 megabyte in size to the normal download location for your browser. When I used Firefox, it downloaded the zip archive to the Downloads folder in My Documents. Find the file using Windows Explorer and double-click on it to unpack all the component files of the archive. I just accepted the default location offered by Windows, which was to create a folder with the same name as the zip archive (vwget-2.4-wget-1.11.4-bin in my case) in the Downloads folder. You can delete the zip file after you unpack it.

A nice feature of VisualWget is that it is entirely self-contained within its own folder. It does not need to be "installed". It does not require any special privileges to use. If you don't want it anymore, just delete the folder.

However, your Windows XP computer must have the Microsoft .NET Framework v2.0 in order to run VisualWget. The simplest way to see if you have .NET installed is to simply try to run VisualWget. Double-click the file VisualWget.exe that you find in the folder of unpacked files. If it runs and opens up the VisualWget window, you are OK. If it gives an error message and does not run, you need to install the .NET framework. You can download it from this Microsoft website. You will need to install it from an account on your computer that has administrative privileges.

Start VisualWget and setup your options

In VisualWget, you first create a download definition that has all the information needed to download the files you want from a website. Those definitions are saved by the program (unless you remove them), so they can be reused if the website has changed and you want to update your downloaded files to match.

To create a new download definition, select the New... item in the Downloads menu (this is in the normal location of the File menu). You will be given the option of simply typing the URL of a single web page to download, or defining a set of options for your download. In order to download an entire website, you need to define options, so select that.

Warning: while you are setting options in the New Download window, do not click the "OK" button. That button immediately starts the downloading process. Only click that button after all options have been set.

The New Download window will open, starting at the General options, as shown in the annotated screenshot below:

Simply type (or cut and paste) the URL of the website you want to download in the "URLs" field (outlined in the red box). Next, tell the program where to setup the directory structure and saved files from this website. Click on the button to the right of the "Save files to:" field (outlined in red with annotated instructions) to bring up the standard Windows open dialog box. Navigate to the folder of your choice and select it. The two options under "Resuming" are normally checked and should be left alone.

Now you need to select some other options that are necessary or useful for downloading an entire website. I will only describe the minimum set of selections you need for downloading a well-behaved website. If you have problems getting the download to work, you should look at the other options. Clicking an option name (not the checkbox next to it) will give a one-line explanation. Since VisualWget is just a front-end to the Wget program, and uses the same option names, you can get detailed descriptions of what these options do from the official Wget on-line manual.

Click on the Advanced field on the left side of this New Download window. This gives you a whole series of sub-sections, each of which has a series of options you can select by clicking on checkboxes. The first sub-section you need to configure is Download. Here are my recommended options for the Download subsection, as shown in the screenshot below.

  • --no-clobber
    Prevents a downloaded file from overwriting an existing file on disk.

  • --continue
    If the download is interrupted, and you restart it, finish getting any partially downloaded files.

  • --ignore-case
    Treat uppercase and lowercase letters the same when downloading files. For example, a jpeg image file could end with either .JPG or .jpg and this option treats them the same.

  • --user=USER
    Optional - check this if the site you are visiting requires a login with username and password. This will probably only work in the case where your browser prompts you for username and password before showing any part of the website. It will probably not work in the case where you have to type in your username and password on fields on the web page (for example, most banking websites). See instructions after the screenshot for how to enter the actual user name.

  • --password=PASS
    Optional - use under same conditions as the --user=USER option, above. They go together.

If you checked the --user=USER and --password=PASS options in the Download subsection (above), you have another step: you have to enter the username and password for the website. You do this by clicking on the option name itself (for example, clicking on the text --user=USER), not on the checkbox. That opens a new field in the lower part of the window where you can type the value needed.

For example, in the annotated screenshot below, I have clicked on the --user=USER text, and that gives me a USER: field where I can type the username needed to login to the site. The second screenshot shows the same thing for the --password=PASS option.

Now, you have to set the options that tell Wget to get the entire contents of the website, not just the one page in the URL you specified.

Click on the Recursive Retrieval subsection heading of the Advanced options. Here you have to check the box next to two key options, as shown in the screenshot below.

  • --recursive
    This tells the program to follow links from the first URL, and then links from those links, etc, until it has downloaded the entire website. It will not follow links that go to other domains (off this website).

  • --level=NUMBER
    This option controls how far recursive downloading will be pursued. The original website URL is the first "level" of the site. The pages that it links to are the second level. Within those second level pages are links to third level pages, etc. This parameter controls the maximum number of levels that will be downloaded before the program stops.

    Although the program allows "infinite" recursion, this is not a good idea. A poorly programmed website, especially one with dynamic content, may result in links that point back to other levels, with perhaps very slight changes, resulting in another round of downloads, ad infinitum. It's best to put in a large, but finite, number that should be comfortably larger than the number of levels you expect. For example, in the screenshot below, I use "100" as the number of levels. As for the username and password options, you click on the text of the option name itself (--level=NUMBER) in order to bring up the "NUMBER" field where you can enter the maximum number of levels for the recursive downloads.

    It is also a good idea to do a test run of your website download with this level parameter set to "2". That will verify that it does follow links, but minimize the number of files downloaded for the test.

There is one more option you should set to make sure your download does not get extra files that you don't want. Click on the Recursive Accept/Reject subsection heading of the Advanced options. As shown in the screenshot below, click the box next to the --no-parent option. This option tells the program not to follow links that go back up to a higher level of the website. For example, if you are downloading, this option will prevent the program from following links that go back up to the parent site at

Run your download job

Following the instructions above, you have configured the options for a website download job in VisualWget. Now simply click on the OK button in this New Download window to start the downloading process.

VisualWget saves your download job definitions. You can re-run the job later, perhaps with modified options. To edit the definitions before re-running, you can right-click on the definition name in the main VisualWget window, and then select the Edit item, as shown in this screenshot:

Comments or Questions?