... or "Ask Not To Whom The Pointer Points, It Points To Thee."
TL;DR: A pointer is not a reference. A pointer knows nothing about the data
being pointed to. Returning multiple values requires actual work.
Everything went wrong when I wanted a string with a NUL character inside it.
C strings are not Perl scalars, they don't know how long they are. So to mark
the end of a string, C uses the NUL character, \0. The strcpy function will
copy to your destination until the first \0 from your source. When you want to
have a string with a \0 inside of it that does not mark the end of the string,
you need to know exactly how long the string is. This is not difficult to do,
but you also have to return that length from the function that creates your
string.
C functions do not have more than one return value.
(char* buffer, int bufferSize) = get_string_with_nuls();
// You thought it could be that easy?
So in order for your function to result in more than one value, you have to
pass in references to be used to fill in with actual values.
char* buffer;
int bufferSize = get_string_with_nuls( buffer );
// C programmers will already know what I did wrong here
Thinking like a Perl programmer, I thought I could just pass in the pointer to
the function and the function could fill it with data. Two problems:
- I passed in the pointer itself, not a reference to the pointer:
&buffer
- I did not initialize the pointer to anything.
A more correct way would be:
char* buffer = malloc( 128 * sizeof( char ) );
int bufferSize = get_string_with_nuls( &buffer );
But this suffers from another problem: I have to know beforehand how big my
string is going to be and allocate that much memory beforehand.
The way I ended up succeeding was:
int bufferSize;
char* buffer = get_string_with_nuls( &bufferSize );
This way, get_string_with_nuls can handle the malloc with exactly the correct
size and give it to me. I don't have to guess at a size beforehand.
Of course, a struct could do this better, or since I'm actually in C++, an
object. I'll be planning a new API as soon as I confirm this one actually
works and has proper tests (written in Perl, of course).
The Server Is Down
- No it isn't, I didn't get paged.
- Wait a minute, why didn't I get paged?
- FUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU--
- CHALLENGE ACCEPTED
Diagnosis
The client reported that the site sometimes took more than a minute to load.
Doesn't respond very slowly to me, and the pager is only primed to ping me if
there is a sustained downtime (hiccups are not something I want to wake up for
every night at 3:00am).
Strangely, load hovered around 7 most of the time, only spiking to 13 every
few minutes. With a 16-core processor, this was well within operating
parameters, if just a little worrisome. Nothing in the log files.
Oops, now I get a slow page load. Takes 30 seconds to load a page. Refresh
again, and the page loads just fine. Clear browser cache, and the page still
loads just fine.
top kept MySQL at the top of the CPU list. Not surprising, as this server is
the master database server for a two node cluster. So I keep an eye on top as
I poll mysql for its process list.
A pattern emerges: The load spikes and server goes unresponsive when this happens:
This table shows 12 different processes are trying to update the same cache
location (process ID 2-3, 5-8, 10, 12-13, 18, 23, and 26). Because of MyISAM's
table-level lock, any request to get from the cache has to wait for 12 REPLACE
INTO requests to complete. They've already taken 1 second, if each replace
takes 2 seconds, that's 24 seconds of non-responsive website.
These 12 processes all saw that the cache item had expired and are trying to
update it. This is called a "cache stampede". Only one of them needs to update
the cache, the rest are just wasting resources. Worse, they're doing all the
work to update the cache, which is much more expensive than getting the value
from the cache. If it's expensive enough, the site goes down hard.
Management
How can we stop the cache stampede? One way is to mildly randomize the actual
expiration date when checking if the cache is expired:
sub is_expired {
my ( $self, $key ) = @_;
my $expires = $self->get_expires( $key );
# Randomize the expiration by up to 5% +/-
# by first removing 5% and then adding 0-10%
$expires = $expires - ( $expires * 0.05 ) + ( $expires * 0.10 * rand );
# Compare against now
return $expires > time;
}
In this very simple case, if you are within 5% of the expiration time, you
have a chance to have an expired cache item. The chance grows as time passes,
reaching 50% at the actual expiration time, and 100% at 5% past the expiration
time.
Rather than add this expiration variance to our custom database cache, I
instead opted to move this site over to CHI, which has this protection
built-in.
my $cache = CHI->new(
driver => 'DBI',
namespace => 'localhost',
dbh => $dbh,
expires_variance => '0.10',
);
This stops the cache stampede, but we're still hitting the database a lot.
Remember we have two web nodes hitting one database node. The fewer database
hits we make, the better performance we can get without having to ask for more
hardware from the client (which takes time, and forms, and more forms, and
meetings, and forms, and more meetings, and probably some forms).
Because this is a distributed system, we need a distributed, synchronized
cache. We cannot use memcached, as WebGUI 7.x does not support it (but WebGUI
8 does). So for now we must use the database as our synchronized cache, but
what if we put a faster, local cache in front of the slower, synchronized
cache?
CHI has an awesome way to do this: Add an l1_cache
my $cache = CHI->new(
driver => 'DBI',
namespace => 'localhost',
dbh => $dbh,
expires_variance => '0.10',
l1_cache => {
driver => 'FastMmap',
root_dir => '/tmp/cache',
},
);
Now we're using FastMmap to share an in-memory cache between our web
processes, and if the L1 cache is expired or missing, we look for content from
the DBI cache. If that cache is missing or expired, we have a cache miss and
have to recompute the value.
Hurdles
I had to install the DB tables myself, which was not difficult, just
undocumented (bug report filed). MySQL only allows a 1000-byte key, and the
CHI::Driver::DBI tries to create a 600-character key. This is fine in the
Latin-1 charset, but MySQL complains if you're using UTF-8 by default.
The driver also tries to create a TEXT field to hold the cache value, but
MySQL expects a text field to hold characters in a known character set. After
noticing that my cache values were empty, I changed to a LONGBLOB.
The full create table statements are below:
-- primary cache table: chi_<namespace> --
CREATE TABLE IF NOT EXISTS `chi_localhost` (
`key` VARCHAR(255),
`value` LONGBLOB,
PRIMARY KEY ( `key` )
);
-- CHI metacache table --
CREATE TABLE IF NOT EXISTS `chi__CHI_METACACHE` (
`key` VARCHAR(255),
`value` LONGBLOB,
PRIMARY KEY ( `key` )
);
Results
The server is stable again! Spikes do not turn into out-of-control loads and
unresponsive server. We'll see how things go tomorrow during normal business
hours (the peak time for this site), but right now it looks like CHI has saved
the day!