Dealing with WHOIS records

Published: January 4, 2015

Recently I've been working on a side project, Domain Clamp, a web app for monitoring domain and SSL expirations and sending notifications. The most unexpected challenge thus far has been dealing with WHOIS records. While SSL expiraton dates can be retrieved simply by asking the server on which the SSL is installed, there is no way to get a domain expiration date without asking a WHOIS server. And that, my friends, is where the trouble begins...

Some Background

I initially began building Domain Clamp in Rails, but quickly switched to Laravel. I'm, sure Rails would've been an excellent choice for the project, but I'm much better versed in PHP and wanted to be able to hop right into the development without the learning curve of a new language. There were two critical requirements for the application

The ability to automatically check domain expiration dates
The ability to automatically check SSL expiration dates

SSL expirations turned out to be by far the easier of the two. Some quick Googling turned up this stack overflow thread and I wound up creating a simple Laravel package. It's really just one method and could even be placed in the app's helpers file. Here's the code:

function getCertInfo($domain)
{
	$c = stream_context_create(array("ssl" => array("capture_peer_cert" => true)));

	set_error_handler(function(){return true;});
	$r = stream_socket_client("ssl://".$domain.":443", $errno, $errstr, 30, STREAM_CLIENT_CONNECT, $c);
	restore_error_handler();

	if(!$r) {
		return false;
	}

	$cont = stream_context_get_params($r);
	$certinfo = openssl_x509_parse($cont["options"]["ssl"]["peer_certificate"]);
	return $certinfo;
}

Data on the certificate is returned as an array and $certinfo['validTo_time_t'] contains a unix timestamp with the certificate expiry.

TLD Administrators and WHOIS records

At the time of writing this article there nearly 800 active TLDs. Conveniently, a machine readable list, is maintained by IANA (the Internet Assigned Numbers Authority). Operation / administration of each TLD is split amongst hundreds of different entities. IANA also maintains a webpage which details the operator for each TLD. Unfortunately, there is no standardized response format for a WHOIS request. Per the WHOIS spec

The protocol delivers its content in a human-readable format.

Human-readable is the important word here...Translation, your app cannot get the domain expiry from a WHOIS response without some extra effort.

Getting the Domain Expiry

Back to Domain Clamp, as mentioned above, automatically pulling a domain expiration date is one of the core requirements of the application. Rather than re-invent the wheel, I reviewed existing libraries that work with WHOIS records. Essentially there are two steps involved.

Step 1: Determine the proper WHOIS server for the query: WHOIS libraries generally contain a configuration file mapping TLDs to WHOIS servers. Here are a few examples:

Step 2: Parse the response from the WHOIS server: WHOIS libraries use RegEx to parse the hundreds of possible responses. The parsers directory from the Ruby Whois Gem currently has over 200 files in it.

How I'm Handling This For Domain Clamp

Since I made the decision to build the app on top of Laravel I naturally began to look for a PHP solution. There are a number of libraries available which range from providing virtually no parsing functionality to providing a fairly solid API for interacting with the WHOIS record. Initially, I selected the Novutec library. Things started out smooth, but as I began to test some of the domans I own, I found many cases where the library was not returning the proper WHOIS result.

After submitting a PR which went unacknowledged for several months, I toyed with the idea of maintaining my own fork of the library, but eventually would up deciding to set up a separate service for WHOIS queries built with the Ruby WHOIS gem. The service is set up as a Sinatra app and I decided to throw it up on GitHub.

How Could This All Be Improved

This post is not meant to point a finger at Novutec. They put together a solid library which I'm sure took countless hours to develop and have open sourced it to the community asking for nothing in exchange. WHOIS parsing is hard and is even harder to maintain.

Ideally, the WHOIS spec would be updated to outline a standard format for WHOIS server responses. Then IANA, who is responsible for management of the DNS root zone would be in charge of enforcing that TLD operators comply with that standard. Alternately, IANA could provide it's own WHOIS server in a standardized format.

Knowing that the above is unlikely, at this point everyone who has a stake in WHOIS record parsing needs to actively contribute to libraries to ensure they stay up to date

A Note On Testing

Aside from the above, another gripe I have with WHOIS is around special use domain names. The following second level domains are all currently off limits according to RFC 6761.

example. (e.g. example.com)
test. (e.g. test.com)
invalid. (e.g. invalid.com)
localhost. (e.g. localhost.com)

However, many TLD operators do not return a standard WHOIS response for these reserved domains. This means that there is not universal second level domain that can be tested across all TLDs to confirm that a WHOIS parser library is parsing records properly. Here's an example of how simple it would be to test the Novutec library if test. was mandated to return a standard WHOIS record from the WHOIS server.

require_once 'DomainParser/Parser.php';
require_once 'WhoisParser/Parser.php';
$Parser = new Novutec\WhoisParser\Parser();

$tlds = parse_ini_file('WhoisParser/Config/whois.ini');
foreach($tlds as $tld => $data) {
        $domain = 'test.' . $tld;
        $result = $Parser->lookup($domain);
        if(!$result->expires) {
                echo $tld . ' did not return an expiry date' . PHP_EOL;
        }
}

The Future

The WHOIS system is widely regarded as broken. As an app developer I may gripe about the lack of a standardized response format, however ICANN has questioned the fundamentals of giving out anonymous public accss to registration data. It will be interesting to see the future of WHOIS. From my viewpoint, I can only hope that the system becomes easier to work with for developers using the data for "permissable purposes".