Moving to GitHub, slowly

The software on this pages will slowly be moved to GitHub https://github.com/hilbix/. The CVS repository will be migrated to GIT as well, so the history will be preserved, a bit. See FAQ.

Scylla and Charybdis, dbm - Tools

The tools are developed under Linux with ESR's paradigm release early, release often in mind.
So you can consider this beta software, or alpha, or pre-alpha, or even worse ;)

Have a look in the download directory for all downloads.
As always here, all you get is the source. No binaries here.

dbm 0.11.0-20080519-114741

A little tool to access gdbm files from shell.

⇒ ⇒ ⇒ The development shifted to another location. All future versions will be published at https://github.com/hilbix/dbm ⇐ ⇐ ⇐

If you have an Online Virus Scanner under Windows, never use options -t, -q or -a. With these options and a Virus Scanner you will observe strange things to happen. Under Linux (or Unix) there is no such problem.

If a DBM process cannot lock a DBM file, it enters a rapid open/close busy cycle (GDBM is lacking support for blocking locks) until the timeout occurs. This triggers a race condition with Online Virus Protection Solutions for Windows such that it becomes unlikely that DBM can get a successful lock on the DBM file.

Note that DBM works as designed, this is no bug of DBM. However the incompatibility in the Virus Protection cannot be fixed without rendering them nearly useless. So the bug is in the idea of Online Virus Protection. This means: Under Windows you currently cannot have both: DBM processes concurrently accessing a DBM file and an Online Filesystem Antivirus.

If only a single DBM process accesses a DBM file (so you do not need a timeout option at all), then you do not need to worry about.

Note that DBM is deprecated, but I have no time to create a replacement

DBM began as a quick hack just to access gdbm files from the commandline. And it still is evolving as a quick hack.

It now has some batch and filter functions to be used in shell pipes, but it is very little tested for now.

When you compile it, just ignore the error that make cannot cd into "tino". That's ugly, but as it's just a quickhack no tinolib shall be required to compile it, and thus the current version of tinolib is not part of this distribution. Also note that there currently is no tinolib distribution, as it's far from beeing ready. For now it just comes with other of my sources.

Short example what DBM can do for you, a JPG file checker:

./dbm create db.ok
./dbm create db.fail
find / -type f -name '*.jpg' -print |
./dbm filter db.ok |
./dbm filter db.fail |
sort |
while read -r a
do
if djpeg "$a" >/dev/null
then
./dbm insert DB/ok "$a" 1
else
./dbm insert DB/fail "$a" 1
fi
done

The "sort" is a hack to "buffer" the output, such that the database files are closed when the loop runs such that the insert can be writer. You don't need this anymore if you give '-a-1' to each of the dbm commands.

History:

version 0.11.0-20080519-114741

download (30744 bytes) sig

Functions sadd and sdir removed for re-write

The "find" function now returns FALSE in case it does not find anything. This is to more easily differ from the case when it returns a blank result, which, from shell, does look alike.

The "sadd" and "sdir" functions have proven badly designed, so they will be re-invented with another algorithm. This is an intermediate version which removes the old "sadd" and "sdir" function. If you need the old behavior, stick to version 0.10.x

version 0.10.1-20080407-040731

download (31239 bytes) sig

Improved: Option -a Added: Option -n and -u

Option -a still was too slow on outputs. Now output is processed in "batches", too.

Option -u flushes output as soon as meaningful. Together with Option -a this nearly brings back the old behavior (db close on each line).

Option -n suppresses fsync() on the GDBM file. Changes in the DB are factors faster on machines which are heavily loaded with IO. Especially useful on option -a and batch commands. However the DBM is not synced anymore, so a sudden power loss can have a more bad impact then it already has (often I observed that a sudden power loss corrupts the DBM).

version 0.10.0-20070806-101555

download (30382 bytes)

Improved: Option -a Started to add "Sorted keys directory structure"

Option -a was very slow. Now input is processed in "batches", that is the database is closed only between actual read() calls.

Also new experimental "sadd" and "sdir" commands introduced. "sget" and "sfind" are still missing.

version 0.9.2-20070407-210740

download (28935 bytes)

Bugfix: Now exit(11) in case that the output pipe goes away.

When output is futile, DBM shall go away, as further processing is no more interesting (as nothing which outputs data does not alter data).

And to stress it again:

Currently I use DBM in a fairly complex environment with several dozens of scripts running parallely on a handful DBM files concurrently. So I am was able to hunt down some really weird race conditions and pitfalls. I am now pretty sure that I cought the most. But one thing cannot be fixed:

Read the virus scanner warning when using DBM under CygWin. If you have several concurrently running processes under Windows which use DBM on the same file, you will see that they either starve all or a lot of DBM processes start to pile up until your system becomes totally unusable. In this case do following:

Disable the online virus scanner.
Set the priority of the DBM processes to lowest possible value.
Kill the processes, each. After some hours they might go away.
Uninstall the online virus scanner component and reboot your PC.

Again, this is no fault of DBM. DBM works correctly as designed. It has no means to detect or circumvent such a case where online virus scanners lock files for some time after they were closed or opened by an application. DBM only(!) tries 10 times a second to get a lock on the DBM file. Note that this rate is *low*, as you need N second timeout for N processes concurrently accessing the file with this low rate to have a high enough chance to get the lock on the DBM file. Now, if a virus scanner is able to scan the file 100 times a second and you have 5 DBM processes concurrently trying to lock the file, this means that you have a 50:50 chance that you get into trouble.

version 0.9.1-20070326-202709

download (26860 bytes)

Bugfix: balter did not work (bug in db_cmp() removed)

[view more history]

[view complete history]

License and Disclaimer

All you can see here is free software according to the GNU GPL.
Copyright (C)2000-2011 by Valentin Hilbig
Note that the software comes with absolutely no warranty of any kind.
You use the software at your own risk.
Valentin Hilbig cannot be hold responsible for any unintended damage,
lost data or malfunction of the software you can find here.

Last modified: 2011-09-12 by Valentin Hilbig [ Imprint / Impressum ]