Free Software for data analysis, UNIX administration and Web automation

All the programs that you find here are Free Software which can be redistributed and/or modified under the terms of the GNU General Public License of the Free Software Foundation (FSF). They are written in script languages (shell, perl and expect), and should run on most UNIX-like systems. Technical documentation to some of the projects, and also a howto about "Connection load-balancing of PPP-Slirp serial links over SSH using EQL" are available here.



Program/Package Description Remarks
Use CopyLeft to support software freedom.
Diligence shell

dish
v. 1.19.1
deb
rpm

checksums
The diligence shell 'dish' executes commands via ssh/rsh/telnet/mysql/... simultaneously on several systems. Changing password, updating configuration, copying files, checking status, etc. on hundreds of nodes is made then simple. Dish is similar to the IBM's 'dsh' (distributed shell) but furthermore allows remote execution of commands requiring authentication. Compared to other open source packages which include a distributed shell (or similar tools) like ClusterIt, dancer's shell, parallel distributed shell, or C3, dish is a slim and easy to use remote-shell wrapper for cases where a flexible solution is needed. For more details you can check the manual page. Requires expect, tcl and ssh (alternatively rsh or telnet);
For 'dish' to work in parallel mode, Tcl should be compiled without "--enable-threads".

This package is included in the Debian GNU/Linux distribution and the Free Software Directory of FSF.
Simple Tools for Data Analysis

stda
v. 1.2.1
deb
rpm

checksums
The package includes some primary tools for data analysis - you can evaluate sums, averages, integrals, histograms, and distributions of 1-d data, and eventually plot the results. The separate programs are stand-alone tools (supporting the standard UNIX input and output pipelines) to be used for data processing from the command line. Requires gnuplot and awk;
STDA contains some of the scripts listed below (maphimbu, mintegrate, muplot, and prefield).

This package is included in the Debian GNU/Linux distribution.
maphimbu
v. 2.8.4
Histogram builder from numeric or text data input - it can compute histograms and probability density functions of 1-d data. You can use it for logfile analysis (of number of visitors, requests, downloads, etc.), performance evaluation and others. Requires awk;
Included in STDA;
mintegrate
v. 2.2.1
Evaluates average, sum/integral and derivative of 1-d numerical data Requires awk;
Included in STDA;
lqfit
v. 1.4.5
Gnuplot-wrapper for linear and quadratic fitting Requires awk and gnuplot;
Included in STDA;
muplot
v. 5.3.2
demo
Gnuplot-wrapper for instant viewing and printing of plots. It has a very simple command-line interface. With this fast tool for producing plots you can render multiple data files on a single plot at once. Just execute 'muplot -s "*.dat"' and a post-script file of the plot will be created and presented on your display. Requires awk and gnuplot;
Included in STDA;
You can look at the demo with examples of how 'muplot' and 'prefield' work - simply unpack the archive and start "./demo".
prefield
v. 1.4.8
Prepares suitable input for 'muplot' to plot 2-d fields with arrows
nplots
v. 2.2.2
Arranges multiple PostScript figures (files) on a single A4-plot Requires LaTeX2e and epsfig ; Ghostscript is recommended
.urlmon.mfs
v. 1.6.3
A general purpose content filter for urlmon which produces reports and summary of reports from www-monitoring. In the simplest case, it reports differences between previous and present text content of a monitored www-page. In advanced use, the filter can parse a monitored page, make a list of included url's, then based on regex match, fetch and convert some of them into ascii text, eventually filter the text and send a report per e-mail. For example, one can set up an alert system for www-monitoring of news, job postings, price discounts offers, etc. The code is written in Perl. An example for applying '.urlmon.mfs' on an URL-database can be found in this tar-gz-archive including the urlmon (ver. 4.0b) distribution with a slightly modified urlmon (ver. 4.1.1) - it enables perl-variable-style for paths in the database file and also uses sha1 checksums. Unpack the archive by "tar xzpvf urlmon.tar.gz" and change directory to the one created after unpacking. After executing "./urlmafor ." you should receive 3 e-mails with reports from the monitored pages put in the example database. You can view the documentation by executing the command "perldoc .urlmon.mfs".
urlmafor
v. 0.5.2
Executes urlmon in forking mode on a list of database files. It allocates automatically and individually for each database the number of processes to be forked.
mkbook
v. 1.2
Makes an A5-booklet from a PostScript file Requires psutils;
Part of the code was originally written by Ljubomir Milanovic.
rpmoncd
v. 2.2
Collects files or rpm-packages from locally accessible dir(s) and prepares a single CD/DVD-distribution Requires mkisofs if you want to create ISO-images from the prepared distribution-directories
Last Modified 21-Jan-2014 by admin