BashDB, a single dynamic database table for bash scripts

I write shell scripts. A lot of them. I automate everything with scripts. Many of these scripts need to have some notion of a state. The state needs to be persistent. How do we do that? And could we draw some retro flashback line character table borders while we’re at it?

The problem

If I have some sort of a watchdog running every 5 minutes, I may not want to get an email about it until it has failed 12 times, or continuously for 60 minutes. I could run my script continuously and just sleep, or I could store the “fail count” in some file and read it back in on the next iteration. I often opt for the latter, and I used to find myself re-implementing pretty much the same tab separated, line separated, or null separated storage algorithms over and over.

Then come the lists. Say I have a script that’s supposed to verify that a couple of URL’s are online. Where do I store those? One per line in a file? I name it script.conf and put it somewhere? That’s fine, but what if I want to also store the last time each URL failed, so I can include more information in a report? Tab separated “columns”? A different file for the dates? Should I use a simple database, such as SQLite? That seems like a bit of an overkill, and it might not be installed where the script is supposed to be running.


Enter BashDB, written in the car going to and from a weekend trip at the cottage. It stores a single table, with a simple Key=>Value structure, or named columns. It dynamically adds new columns as requested, supports list values in any column, supports binary data, and it’s all written in Bash with very few external dependencies.

It can be downloaded –> HERE
View the manual –> HERE  (or run db_help after sourcing the library)


To use bashdb, simply source it in the top of your script and use the provided db_* functions:

db_help The equivalent of running each function, in turn, with the -? option. Shows the help 🙂
db_columns Lists the columns currently in the table
db_copy_row Copies a row with a given key to another row, optionally overwriting it if it exists
db_delete Deletes a row with a given key
db_delete_column Deletes one column of data from all rows
db_dump Provides a somewhat pretty display of the data currently in the table
db_get Gets a single value from the table
db_has_key Returns whether or not a given key exists
db_has_column Returns whether or not a given column exists
db_keys Lists the keys in currently in the table
db_rename Changes the key of a row
db_rename_column Renames a column
db_search Lists keys or dumps data matching a search regex in a given column
db_selftest Runs a unit test of most features
db_set Sets a value on a key
db_testdb Generates a small test database
db_trim Removes any empty columns


To try out the functionality, one can also use the script from the console, as such:

$ source ./bashdb
$ ls test
ls: cannot access test: No such file or directory
$ # we don't have a "test" yet :)

Adding a key and a value:

$ db_set
Error: db_set: Missing parameter: -f <file>
$ db_set -f test
Error: db_set: Missing parameter: -k <key>
$ db_set -f test -k Key1 -v Value1

Adding some more keys, and changing the value for key1:

$ db_set -f test -k "Norwegian Characters" -v "Ææ" -v "Øø" -v "Åå"
$ db_set -f test -k key1 -v "Value2"
$ db_set -f test -k key9 -v "Day[9]"

The “Norwegian Characters” is an example of storing a list. Lists can be read back out, either “human readable” -h, or null byte (\0) separated:

$ db_get -f test -k "norwegian characters"
ÆæØøÅå$ db_get -f test -k "norwegian characters" -h
$ db_get -f test -k "norwegian characters" -h | while read -r c; do echo "--> $c <--"; done
--> Ææ <--
--> Øø <--
--> Åå <--

We can unset values by setting a value to an empty string, “”, or by simply not including the -v switch at all, when calling db_set. A row will be removed entirely if all its values are unset.

Lists (which are actually any value at all), support the -m switch, followed by “add” or “remove” to modify the list:

$ db_set -f test -k key1
$ db_set -f test -k "norwegian characters" -v "Øø" -v "Åå" -m remove
$ db_set -f test -k key9 -m add -v "Day[10]"

Without specifying -c, a default unnamed column is used for the data. However, columns can be added dynamically to include more information about each key:

$ db_set -f test -k key9 -c "Additional" -v "Test Value"
$ db_set -f test -k key9 -c "Additional 2" -v "More stuff" -v "Here"
$ db_set -f test -k new -c "Additional 2" -v "Not all columns need to be filled"

Column data can be requested by using -c on db_get. Also showing an example of -d, giving a default value if none is set, as for key=>new, column=>additional:

$ db_get -f test -k key9 -c "additional 2" -h
More stuff
$ db_get -f test -k new -c "additional" -h -d "This is the default"
This is the default

Taking a short break to run the unit tests:

We can easily delete a row, or an entire column:

$ db_delete -f test -k new
$ db_delete_column -f test -c additional

Listing keys and columns, also with the -h option, since we’re showing it here.
The default column shows as an empty line.

$ db_keys -f test -h
norwegian characters
$ db_columns -f test -h

additional 2

We can also store values from stdin, for instance a web server response. This even works with binaries, though I wouldn’t recommend trying db_dump on tables containing those. Not that it breaks, it just looks like crap.

$ curl "" | db_set -f test -k "Response from" -i
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 269 100 269 0 0 5458 0 --:--:-- --:--:-- --:--:-- 5489
$ db_get -f test -k "response from"
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<H1>302 Moved</H1>
The document has moved
<A HREF=";dcr=0&amp;ei=ej7bWYqlOuLk8AfSmqqoDg">here</A>.

When dumping a table for display, values are truncated. Please note that if you’re storing binary data, the displayed length by db_dump will only include the printable characters, and will not accurately reflect the length of the binary that would be retrieved by db_get.

Searching is done using a regex on a given column, optionally running db_dump on the results instead of listing the keys:

If you decide to use this for your own projects, and you’re having issues, please read the output of db_help first, or run the function you’re having issues with using the -? switch. Or simply ask me here.

Have fun 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *