Useful Tools for SWAT Modeling

SWAT requires large amounts of input, and produces large amounts of output. The GIS interfaces (AVSWAT and AVSWAT-X) do a great job of setting up the input files. And, the output files are well-organized—but large and cumbersome at times. Much of the useful output is summarized in the output.std file. However, we've found that we often need very specific summary data from selected points in the watershed, and at different points along the flow path from the uplands, down the channel, and finally at the watershed outlet. For this, we've found that some additional skills in programming and data base manipulation have been needed to efficiently extract the most useful data for specific needs. So the following is a list of auxiliary software that we have found useful. Reference to trade names is not meant as a specific endorsement; there may be better software out there, but this is what we have used:

Basic data analysis and manipulation

Microsoft Excel
Clearly a spreadsheet program is useful for managing data, for both input and output. Like most folks, we use Excel, which can read and write dbf files that can be read by ArcGIS. Excel has some quirks, and occasionally data that displays fine in Excel is for some reason not properly readable by ArcGIS or SWAT. Consequently for some SWAT input files, we might generate the data in Excel, but then export a text version (perhaps as a csv file) for input into SWAT.

It's also useful to learn enough Visual Basic for Applications (VBA for Excel) to make some of your own functions—such as the Nash-Sutcliffe Coefficient of Efficiency, for example. I bought a small introductory guide from a local bookstore; perhaps the online help for VBA will be enough for you, but I found the information just a bit over my head to start with.

Excel quirk—Changing the number of rows and columns in a dbf file is not so simple.

If you read a dbf file into Excel, and add a row or column, and then save the file again as a dbf, Excel will ignore the added (appended) rows and columns. This is because when Excel reads a dbf file, it gives the data range a name ("Database") and extent (number of rows and columns) for the original data. If you want to add a row or column, you'll need go to the menu and select "insert," "name," "define," select "Database", and then increase the range extent for the new data range. If instead you insert rows or columns within the original range, then Excel will expand the data base range automatically. But appending rows and columns outside the original range require the extra step of manually increasing the named range's size.

NotePad, WordPad, or other text editor (including Microsoft Word)
Clearly at times you'll need a simple text editor. We prefer the basic Notepad for most files, rather than a full word processor like MS Word, simply because of the reduced chance of hidden formatting commands becoming embedded in the output file. Presumably saving files as "text only" in MS Word should be safe. Maybe.

GNU Emacs
Emacs can be used as a text editor as well. I use it from time to time simply because (when in column mode) it gives you the column number of the cursor location, which can be useful in trying to figure out the column location of SWAT output data. This is essential information when writing small FORTRAN programs to extract and process desired data (see below). Emacs is, however, much more than a simple text editor; it has many adherents among the computer savvy. I find it clunky, but that's because I use it only a few times each year for very limited purposes, and I don't have the cursor-control commands memorized.

Microsoft Access
We've found Access (or some other database management system) to be tremendously useful in extracting useful data from SWAT output files. Usually, because of the time to write the output files for daily or monthly output, we usually only write output files (which are dbf files) for annual output. Using Access to analyze subbasin output files (like sbs.dbf in the tablesout directory), we can summarize sediment and nutrient yields for specific crops, or for HRUs with specific crop rotations, for selected time periods. We often need additional dbf files (created in Excel) to add identifying information to help extract the exact data we're looking for from the large subbasin output files. But overall, this is much more efficient than trying to use Excel alone to sort the output and extract just those specific lines of data you need.

We also use Access to modify the input dbf files as well. We might want to modify the CNOPs for just selected crops or rotations in the management files. With Access we can update the specific HRUs we desire, efficiently. (Some of this might be possible to do as well with the field calculator from within ArcView.)

Finally, Access seems to write "cleaner" dbfs than does Excel. For whatever reason, sometimes Excel-generated dbfs are not readable by ArcView (though this is generally uncommon); we've had very good luck with Access-generated dbfs, which seem to be always readable.

My biggest complaint about Access is that if you write your queries directly in SQL (which I enjoy trying to learn), Access will not let you comment your code. Commenting code is how I preserve things I have finally figured out; without comments I waste time re-solving problems and trying to remember why I had done things a certain way.

ArcGIS and ArcView3
I suppose it goes without saying that you will need to learn enough GIS software methods to help process your spatial data, both to create input files for the AVSWAT interface and for visualizing the spatial distribution of some output variables. It would be useful to have a list of common tasks and methods that are needed by most SWAT modelers.

I realize there is an effort to develop a GIS interface for SWAT using open-source GIS software, which is an extremely valuable contribution given the annual fees associated with the ESRI products.

Programming

Force FORTRAN
Writing small FORTRAN programs to process SWAT output has been extremely helpful. You can get a simple, efficient, and free compiler at from the Force FORTRAN webpage at http://www.guilherme.tk.

We haven't done more than write simple one or two page programs that read a selected SWAT output file, store the desired variables in an array, process or summarize those data, and produce output. Sometimes the output is just a written to the command console (which can be directed to a text file), and sometimes the output is a data file we can import into Excel.

We use such programs to summarize output that otherwise would take forever in something like Excel. Commonly we summarize monthly or daily output data with these programs. (As mentioned above, we often use Access to summarize annual output files. Access could probably be used for monthly or daily data as well, but AVSWAT takes so long to write these output dbfs that we just don't like waiting.)

Writing simple FORTRAN programs is pretty easy, and that's all you need for most tasks. You'll need to learn how to open files, read lines from those files (skipping the lines you don't care about), read and store selected variables in the lines of interest, process those variables to produce the desired summary data, and then write the results either to the command console (stdout) or to a text file. One or two example source-code files that exemplify these commands, along with a small test data set, should get you well on your way. To that end, here is a tiny FORTRAN source code file and data file as an example of how to extract selected data out of a SWAT output file:

• reachexample.f (the source code file)
• basins.rch (a SWAT output file from our watershed used by this program; ~1 MB)

Certainly other programming languages could be used to do the same thing (see below). I suspect that PERL and perhaps Python are also good at these things. But SWAT output tends to be well-organized in column format, which is exactly what FORTRAN expects (this is not surprising, since SWAT is a FORTRAN program itself!). Plus, learning a little FORTRAN can go a long way in helping you understand how SWAT works, if you venture into reading the source code files.

Other FORTRAN compilers
It may well be that other free compilers are as simple to use as Force FORTRAN; we just haven't explored them. If you ever get into modifying the SWAT source files themselves, perhaps the way to go is to investigate the GNU project compiler, gfortran. I suspect this would be necessary if you were to need to link multiple source-code files (which I'm not sure Force FORTRAN can do—perhaps it can). Intel makes a full-featured compiler, but you will have to pay for it; my preference would be to support the open-source software solution.

Java
Java is (to my mind) much more complex than FORTRAN for the simple tasks we usually do. But at some level it then becomes more adept. It seems to be the language du jour, and it's probably worthwhile to learn something about it. I am not yet functional in Java, but my graduate student is keen on it for more complicated tasks than described above in the FORTRAN discussion. Most computers now come with some sort of Java virtual machine and compiler installed. You can download updated versions from Sun. Borland offers a free integrated development environment (IDE) named JBuilder, which we've used. As mentioned above—PERL and Python are well liked by many people. I have no experience with them.

Visual Basic
As mentioned above, learning a little Visual Basic for Applications (VBA) is well worthwhile for extending Excel's capabilities. Although we use it mostly in Excel, others have created macros in MS Word to process data input files. And, it is now used in ArcGIS for some tailored operations. My advice is to learn enough to write your own functions in Excel, as needed. Then, when you find an ArcGIS routine on the web that gives the needed Visual Basic code for some specific application, you won't be intimidated. I suppose if you get really good at Visual Basic, you might be able to use Excel for everything that we use our little FORTRAN programs for—and possibly what we use Access for. But my advice would be that learning a little something about how database managers work (and a little SQL) by using Access is still a valuable tool that will be applicable to other aspects of your work beyond SWAT—the concepts and terminology are ubiquitous in so many of my projects that require large data sets and geographic information. Besides, VBA is useful within Access as well as in Excel, though I haven't used it in that capacity.

Statistics and graphics

R
The R statistics package is extremely versatile with equally flexible graphics capabilities. This versatility and flexibility translate into a fairly steep learning curve for data manipulation, processing, and visualization. But the user community is large and support is robust. If you've got to learn a statistics package anyway, this is one of the best to learn. It will take some time to get used to—but it should be around for a long, long time and what you learn now will serve you long into the future (at least that's the general consensus).

HOWEVER, if command line programming is something that turns you off; if you don't have the passion and patience to learn yet another programming language—then you're probably better off going with a commercial statistics and/or plotting software package. I still use SigmaPlot or DeltaGraph for most of my plotting needs. Perhaps someday I'll get good enough at R to give them up—but not quite yet.

Adobe Illustrator
We use this to draw basic diagrams and to dress-up maps and plots for publication. It's an industry standard, but expensive and somewhat complicated. An alternative is CorelDraw, which (I believe) is cheaper and also well-regarded.