The origins of on-call work

| No Comments

On September 6th, Susan Fowler posted an article titled, "Who's on-call?", talking about evolving on-call duties between development teams and SRE teams. She has this quote at the top:

I'm not sure when in the history of software engineering separate operations organizations were built and run to take on the so-called "operational" duties associated with running software applications and systems, but they've been around for quite some time now (by my research, at least the past twenty years - and that's a long time in the software world).

My first job was with a city government, and many of the people I was working with started at that city when they decided to computerize in 1978. Most of them have retired or died off by now. In 1996, when I started there, the original dot-com boom was very much on the upswing, and that city was still doing things the way they'd been done for years.

I got into the market in time to see the tail end of that era. One of the things I learned there was the origins of many of the patterns we see today. To understand the origins of on-call in IT systems, you have to go back to the era of serial networking, when the term 'minicomputer' was distinct from 'microcomputer', which were marketing terms to differentiate from 'mainframe'.

IT systems of the era employed people to do things we wouldn't even consider today, or would work our damnedest to automate out of existence. There were people who had, as their main job, duties such as:

  • Entering data into the computer from paper forms.
    • Really. All you did all day was punch in codes. Computer terminals were not on every desk, so specialists were hired to do it.
    • The worst part is: there are people still doing this today.
  • Kick off backups.
  • Change backup tapes when the computer told them to.
  • Load data-tapes when the computer told them to.
    • Tape stored more than spinning rust, so it was used as a primary storage medium. Disk was for temp-space.
    • I spent a summer being a Tape Librarian. My job was roboticized away.
  • Kick off the overnight print-runs.
  • Colate printer output into reports, for delivery to the mailroom.
  • Execute the overnight batch processes.
    • Your crontab was named 'Stephen,' and you saw him once a quarter at the office parties. Usually very tired-looking.
  • Monitor by hand system usage indicators, and log them in a paper logbook.
  • Keep an Operations Log of events that happened overnight, for review by the Systems Programmers in the morning.
  • Follow runbooks given to them by Systems Programming for performing updates overnight.
  • Be familiar with emergency procedures, and follow them when required.

Many of these things were only done by people working third shift. Which meant computer-rooms had a human on-staff 24/7. Sometimes many of them.

There was a side-effect to all of this, though. What if the overnight Operator had an emergency they couldn't handle? They had to call a Systems Programmer to advise a fix, or come in to fix it. In the 80's, when telephone modem came into their own, they may even be able to dial in and fix it from home.

On-Call was born.

There was another side-effect to all of this: it happened before the great CompSci shift in the colleges, so most Operators were women. And many Systems Programmers were too. This was why my first job was mostly women in IT management and senior technical roles. This was awesome.

A Systems Programmer, as they were called at the time, is less of a Software Engineering role as we would define it today. They were more DevOps, if not outright SysAdmin. They had coding chops, because much of systems management at the time required that. Their goal was more wiring together purchased software packages to work coherently, or modifying purchased software to work appropriately.

Time passed, and more and more of the overnight Operator's job was automated away. Eventually, the need for an overnight Operator exceeded requirements. Or you simply couldn't hire one to replace the Operator that just quit. However, the systems were still running 24/7, and you needed someone ready to respond to disasters. On-call got more intense, since you no longer had an experienced hand in the room at all times.

The Systems Programmers earned new job-titles. Software Engineering started to be a distinct skill-path and career, so was firewalled off in a department called Development. In those days, Development and Systems people spoke often; something you'll hear old hands grumble about with DevOps not being anything actually new. Systems was on-call, and sometimes Development was if there was a big thing rolling out.

Time passed again. Management culture changed, realizing that development people needed to be treated and managed differently than systems people. Software Engineering became known as Software Engineering, and became its own career-track. The new kids getting into the game never knew the close coordination with Systems that the old hands had, and assumed this separation was the way it's always been. Systems became known as Operations; to some chagrin of the old Systems hands who resented being called an 'Operator', which was typically very junior. Operations remained on-call, and kept informal lists of developers who could be relied on to answer the phone at o-dark-thirty in case things went deeply wrong.

More time, and the separation between Operations and Software Engineering became deeply entrenched. Some bright sparks realized that there were an awful lot of synergies to be had with close coordination between Ops and SE. And thus, DevOps was (re)born in the modern context.

Operations was still on-call, but now it was open for debate about how much of Software Engineering needed to be put on the Wake At 3AM In Case Of Emergency list.

And that is how on-call evolved from the minicomputer era, to the modern era of cloud computing.

You're welcome.

Yesterday brought this tweet up:

This is amazingly bad wording, and is the kind of thing that made the transpeople in my timeline (myself included) go "Buwhuh?" and me to wonder if this was a snopes worthy story.

No, actually.

The key phrase here is, "submit your prints for badges".

There are two things you should know:

  1. NASA works on National Security related things, which requires a security clearance to work on, and getting one of those requires submitting prints.
  2. The FBI is the US Government's authority in handling biometric data

Here is a chart from the Electronic Biometric Transmission Specification, which describes a kind of API for dealing with biometric data.

If Following Condition ExistsEnter Code
Subject's gender reported as femaleF
Occupation or charge indicated "Male Impersonator"G
Subject's gender reported as maleM
Occupation or charge indicated "Female Impersonator" or transvestiteN
Male name, no gender givenY
Female name, no gender givenZ
Unknown genderX

Source: EBTS Version 10.0 Final, page 118.

Yep, it really does use the term "Female Impersonator". To a transperson living in 2016 getting their first Federal job (even as a contractor), running into these very archaic terms is extremely off-putting.

As someone said in a private channel:

This looks like some 1960's bureaucrat trying to be 'inclusive'

This is not far from the truth.

This table exists unchanged in the 7.0 version of the document, dated January 1999. Previous versions are in physical binders somewhere, and not archived on the Internet; but the changelog for the V7 document indicates that this wording was in place as early as 1995. Mention is also made of being interoperable with UK law-enforcement.

The NIST standard for fingerprints issued in 1986 mentions a SEX field, but it only has M, F, and U; later NIST standards drop this field definition entirely.

As this field was defined in standard over 20 years ago and has not been changed, is used across the full breadth of the US justice system, is referenced in International communications standards including Visa travel, and used as the basis for US Military standards, these field definitions are effectively immutable and will only change after concerted effort over decades.

This is what institutionalized transphobia looks like, and we will be dealing with it for another decade or two. If not longer.

The way to deal with this is to deprecate the codes in documentation, but still allow them as valid.

The failure-mode of this comes in with form designers who look at the spec and build forms based on the spec. Like this example from Maryland. Which means we need to let the forms designers know that the spec needs to be selectively ignored.

At the local level, convince your local City Council to pass resolutions to modernize their Police forms to reflect modern sensibilities, and drop the G and N codes from intake forms. Do this at the County too, for the Sheriff's department.

At the state level, convince your local representatives to push resolutions to get the State Patrol to modernize their forms likewise. Drop the G and N codes from the forms.

At the Federal employee level, there is less to be done here as you're closer to the governing standards, but you may be able to convince The Powers That Be to drop the two offensive checkboxes or items from the drop-down list.

Resumè of failure

| No Comments

There has been a Thing going through twitter lately, about a Princeton Prof who posted a resume of failures.

About that...

This is not a bad idea, especially for those of us in Ops or bucking for 'Senior' positions. Why? Because in my last job hunt, a very large percentage of interviewers asked a question like this:

Please describe your biggest personal failure, and what you learned from it?

That's a large scope. How to pick which one?

What was your biggest interpersonal failure, and how did you recover from it?

In a 15+ year career, figuring out which is 'biggest' is a challenge. But first, I need to remember what they are. For this one, I've been using events that happened around 2001; far enough back that they don't really apply to the person I am now. This is going to be a problem soon.

What was your biggest production-impacting failure, and how was the post-mortem process handled?

Oh, do I go for 'most embarrassing,' or, 'most educational'? I'd snark about 'so many choices', but my memory tends to wallpaper over the more embarassing fails in ways that make remembering them during an interview next to impossible. And in this case, the 'post-mortem process' bit at the end actually rules out my biggest production-impacting problem... there wasn't a post-mortem, other than knowing looks of, you're not going to do that again, right?

Please describe your biggest failure of working with management on something.

Working in service-organizations as long as I have, I have a lot of 'failure' here. Again, picking the right one to use in an interview is a problem.

You begin to see what I'm talking about here. If I had realized that my failures would be something I needed to both keep track of, and keep adequate notes on to refer back to them 3, 5, 9, 14 years down the line, I would have been much better prepared for these interviews. The interviewers are probing how I behave when Things Are Not Going Right, since that sheds far more light on a person than Things Are Going Perfectly projects.

A Resumè of Failure would have been an incredibly useful thing to have. Definitely do not post it online, since hiring managers are looking for rule-outs to thin the pile of applications. But keep it next to your copy of your resume, next to your References from Past Managers list.

Dunbar's Number is a postulate claiming that due to how human brains are structured, there is an upper limit to number of personal relationships we can keep track of. Commonly, that number is presumed to be about 150; though the science is not nearly as sure of that number.  This 150 includes all of your personal relationships:

  • Family
  • Coworkers
  • Friends
  • People you run into every day and have learned their names

And so on.

This postulate has an intersection with growing a company, and how the office culture evolves. When a company is 4 people in a shared open-plan office, it's quite easy for everyone to know everything about everyone else. You can still kind of do that at 20 people. Getting to 50 starts pushing things, since 'coworkers' begins to take up a large piece of a person's personal-relationship social graph. At 100, there are going to be people you don't know in your company. As it evolves, office-culture needs to deal with all of these stages.

One of the critiques to Dunbar's Number, more of a refinement, is a report by Matthew Brashears (doi 10.1038/srep01513) that claims humans use compression techniques to expand the number of storable relationships. The idea is, in essence:

If the structure of relationships has an external framework, it is possible to infer relationships. Therefore, humans do not have to memorize every leg of the social-graph, which allows them to have more connections than otherwise are possible.

One such external structure has direct relevancy to how offices work: kin-lables.

The example used in the report are things like son, daughter, uncle, father. English is not a concise language for describing complex family structures, so I'll use something it is good at: company org-charts.

Dunbar's OrgChart

If you are a Senior Software Engineer in the Dev Team, you probably have a good idea what your relationship is with a generic QA Engineer in the QA Team. This relationship is implied in the org-chart, so you don't have to keep track of it. The QA team and engineering work together a lot, so it's pretty likely that a true personal relationship may be formed. That's cool.

Scaling a company culture begins to hit problems when you get big enough that disparate functional groups don't know each other except through Company Events. Take a structure like this one:

Dunbar's OrgChart - bigger

When the company is 20 people, it is entirely reasonable for one of the software engineers to personally know all of the marketing and sales team (all three of them). At 75 people, when each of these directorates have been fleshed out with full teams, and both Marketing and Engineering have split sub-teams for sales, marketing, front-end, and back-end, it is far less reasonable for everyone to know everyone else; there is little business-need for the back-end engineers to have any reason to talk to the sales team for any reason other than at Company Events.

This is where the cunning Director of Culture can start building in structure to stand in for the personal relationships it is increasingly impossible to maintain. All-hands Company Events help maintain the illusion of knowing everyone else, at least on a nodding basis. Another way is to start fostering team-relationships across org-chart boundaries using non-business goals, such as shared charity events. This would allow members of the Back-End Team to have a relationship with the Sales Team, which would further allow the individual members of the teams to infer personal relationships with the other team.

This only kicks the can down the road, though.

There will come a time when it will be simply impossible for everyone to know everyone else, even with fully implicit relationships. There will be parts of the company that are black boxes, shrouded in mystery, filled with people you didn't even know existed. A 500 person company is a very different thing than a 100 person company.

As a company grows, they will encounter these inflection points:

  1. Personal relationships can't be held with everyone in the company.
  2. Implicit relationships can't be held with everyone in the company.
  3. Parts of the company are largely unknowable.

As the company gets ever larger, the same progression holds within divisions, departments, and even teams. The wise manager of culture plans for each of these inflection points.

Short version: We tried that, but it doesn't scale. Eventually the one person self-nominated as the sole arbiter of 'good behavior' can't keep up with everyone and has to delegate. When that happens, their subjects start complaining about inconsistent rules and their enforcement.

An extensive, written code allows people to have a chance of living up to that expectation, and understand what'll happen when they don't. This is especially important for volunteer-run organizations who don't have professional, paid enforcers.

Long version

The legal code is just that: code.

Worse, it's a code that is interpreted, not compiled, and how the lines are translated into actions changes a little every time you run through them. Any time the Supreme Interpreter issues a ruling on something, whole swaths of that code will be subject to new interpretations, which will have unpredictable side-effects. Side-effects that will trigger new code to try and handle the edge-cases.

The legal system as it exists in most countries is extensive, written, and impossible for one person to fully understand. This is why lawyers are hated, the legal system seems arbitrary, and anything fun always seems to be illegal. And if that wasn't enough, case-law is its own unwritten thing that handles edge-cases in the lack of a written code. It's no wonder we hate extensive codes of behavior.

That said, there are very good sociological reasons why a code-of-conduct like:

Be Excellent To Each Other

Is a bad code. Take, for example, a basic value judgment:

Murder is bad, and will be punished.

Pretty obvious, and common-sense. And unlike 'be excellent', is narrower in scope. And yet.

Is killing someone accidentally while driving still murder?
Is killing someone in self-defense in your home still murder, or something else?
What is the exact punishment for murder?
Do you punish accidental murders different than intentional ones?
Do you punish killers-of-children different than killers-of-adults?

And so on. This is how we end up with ideas like 'manslaughter' and many grades of murder, with different punishments for each. Without those umpty-hundred pages of legalese defining everything, the answer to all of the above questions would be in case lawwhich is inaccessible to most non-lawyers.

Short codes: Easy to understand in general. But the specifics of what it means to me are completely opaque.
Long codes: Hard to understand in general, but are much more discoverable. If I need to know what it means to me, I can find out.

Nation-states have converged on the long code for very good reasons. But what about volunteer-run organizations like SF conventions, tech-conferences, and open-source projects?

People are hard, let's just party.

Unlike nation-states, volunteer-run organizations like these aren't big enough or well funded enough to have a professional enforcement arm. Relying on peer enforcement is unavoidable, as is relying on untrained people for adjudicating misconduct. These projects can and will attract people quite willing to be enforcers, and are generally the kinds of assholes we want to avoid. The people running these things almost to a person want to avoid conflict, or as it's sometimes called, DRAMA.

If one of your goals is to provide a fun place to code, party, or discuss contentious genre issues, you need a way to bounce the assholes.

Bouncing an asshole is conflict, that thing everyone wants to avoid. When the conflict becomes egregious enough to be officially noticeable, Responsible People tend to respond in a few negative ways:

  • Pretend they didn't see it, in the hopes one of the other Responsible People will notice and do something.
  • Talk themselves into thinking that it wasn't as bad as all that.
  • Pretend they're not actually a Responsible Person, and hope the complainer doesn't notice.
  • Officially notice, but wimp out on actually causing displeasure in the complainant.
  • Hide behind a committee, most of the members of which will be doing one or more of the four previous points.

If you have a "be excellent" code of conduct, point number 2 is where all the Official Drama will go to die; leaving a whole bunch of 'petty highschool bulllshit' to get in the way of the coding, party, or genre discussions. You will have assholes among you, but that's better than being the specific person who ruined someone's day (even if they are an asshole).

If you have a written code with if this BAD then that HARM in it, it allows the drama-avoidant Responsible Person too look at it and say:

Oh shit, this totally applies. Fuckity fuck fuck.

And actually do something about it. Which means, as it was written down, they know what that 'something' is. They may still try to pretend they never saw anything and hope someone else does, but having it written down makes it more likely that the next Responsible Person will do something. It also means that the committee meeting is much more likely to act in consistent ways, and maybe actually bounce assholes.

This is why we keep pressing for details in those codes of conduct. It allows the Responsible People to hide behind the policy as a shield to deflect the displeasure of the punished, and actually provide meaningful direction for the culture of the volunteer-run organization. You deserve that.

Why I'm not moving to California

| 1 Comment

Many companies I would like to work for are based there and have their only offices there, so this stance limits who I can work for. Remote-friendly is my only option, and happily that segment has enough demand I can still find gainful employment in the largest IT market in the US. There are two big reasons for why I won't move to California:

  1. I couldn't stay married if I moved.
  2. The California political system is functionally broken.

Number one doesn't make for a long blog-post, so I'll skip it to focus on the second point.

A failure of democracy: the initiative system

Democracy goes to those who show up.

Government goes to those who show up, are well organized, and have funding.

The initiative process for those of you who aren't familiar with them, is a form of plebiscate. Many western US states have initiative processes, as they were a trendy topic when the western territories were applying to become states. They're seen as a more pure form of democracy than representative democracy, which is what the rest of the US political system is based on. If a group of citizens can gather enough signatures for a bit of legislation, they can get that legislation passed in the next election; in most cases, such legislation can not be overridden by the State legislature.

The intent here is to provide a check on the overriding power of the State legislature, which at the time had a tendency to be captured by industrial interests. Rail-barons and monopolists were a real thing, after all.

With the advent of modern media, a much larger percentage of the population is reachable with relatively little effort compared to the 1910's. In 1910, a special interest (be it a railroad, oil company, or anti-gambling crusader) found their biggest impact on public policy was by lobbying state legislators and other public officials. Best bang for their buck, and didn't require an army of canvassers and hawkers. That didn't stop grassroots organizers from trying to push public policy, they just weren't very good at it; 1914 had 46 initiatives on it, of which 6 passed.

Since the 1910's changes to the initiative process have been made to ensure only initiatives with broad enough public support would be put on the ballot, as voters were getting tired of spending half an hour to vote and way longer in voting-place lines. With modern media, scrounging enough signatures to get a special-interest initiative on the ballot is an intensive advertising campaign away. If an interest can get an initiative passed, the State Legislature can't do anything about it other than live with the impacts.

Democracy goes to those who show up, are organized, and have funding.

Initiative sponsors are the very special interests the initiative process was designed to oust. This leads to initiatives focusing on narrow pieces of government, that over time build a hodge-podge legal system that makes it hard to function.

Raising certain taxes requires a 2/3rds super-majority.
Oh, how about if we ensure budgets have a broad consensus, and require budget-bills be passed with a super-majority.
Education spending is the basis of a healthy population, protect that funding from budget-cuts.
Three felony strikes, and you're outta the public eye for life!
Okay, that's a lot of people serving life sentences, perhaps drug-offenders can get treatment instead.

And so on. It's the kind of code-smell (legal code is still code) that makes you itch to refactor, but refactoring requires going before a committee of managers, some of whom don't know how the system works anymore and are the kind of manager that others need to keep happy.

All of this leads to a government that has to jump through too many hoops, or require unreasonable levels of cooperation between the political parties, to be effective. I don't want to live in a place with government that broken.

There are calls for California to flush it all and rewrite it from scratch have a constitutional convention to deal with this mess, but it hasn't happened yet.

And then there is the Bay Area

To the tech industry, the rest of the state doesn't matter so I'm not going to talk about it.

Did you know that you can do local initiatives too? You bet. But when you get to local politics, only the most invested of interests show up, which selects for existing property owners looking to increase value. Not In My Back Yard is an impulse every city-council meeting has to grapple with. Due to the money involved in the Bay Area, ideas that sound good so long as you're not next door to them get little traction. The few that do end up getting passed face well funded lawsuits by property-owners looking to undo it.

Office-rents in SFO already exceed those of Manhattan. For residential housing, if you can get a mortgage, you're almost certain to be paying more than $2000/mo on it. For renters, over 40% of them are paying more than 30% of their income on the rent (based on US Census data from 2014). Non-census sources suggest rents are even higher, with 2BR units going for north of $3500 on average. To support a housing-payment of $3500/mo, you need to be making $140K/year at least in household-income. For those of us who are single-income families, even Bay Area salaries mean I'll be spending two or more hours a day commuting to work.

Also, San Francisco is the #1 renter-hostile market according to Forbes. San Jose and Oakland take the net two spots. Once you've found a place you can afford, there is zero guarantee it'll still be there in a year or you'll have the same rent.

In the way of big-city in-fill, unit sizes are getting smaller all the time as the available space shrinks.

Impacts to things people generally don't care about, like 401k contributions

There is a funny thing that happens when you make $140K/year, you start running into limits to how you can contribute to your retirement.

If you make more than $132K/year, and are single, you can't contribute to a Roth IRA. But that's a minor thing, since most people are married by the time they hit their 30's, and the limit for household is $189K/year.

The 2016 limit for 401k contributions is $18K. That sounds like a lot, but keep in mind that if you're earning $140K/year, that 18K is only 12.8% of your income. By the time you hit 40, you should be saving 10% or more for retirement. Employer matching contributions can help you get there (and are NOT subject to the contribution limits), but such contributions are few and far between in the startup space, and when they exist at all are not generous.

If you're paying over 30% of income on rent, paying another 10% for retirement is pretty hard.

This is the Federal Government's way of saying:

You will not be retiring in the Bay Area without extensive non-Retirement savings.

Yeah, not dong that.

Nope. I've never lived there. I don't have roots there. Migrating there to chase the labor market is a bad idea for me.

Thanks for reading.

Internet of Patches

| No Comments

This is a good recommendation:

As a sysadmin, I've been saying fuckno to things like Smart TVs and fridges. I do that game professionally, and I know what it takes to keep a fleet of software up to date. It ain't easy. Keeping firmware updated in things like... non-Nest internet attached thermostats (yes, they exist), the PC embedded in the fridge, the hub that runs your smart lighting, the firmware in your BluRay player, internet-attached talking dog toys... It's hard. And it only takes one for Evil People to get inside your crunchy exterior and chow down on everything else.

You can probably trust a company like Schlage to treat their software like a security-critical component of a network. You probably can't say the same about the internet-attached talking dog toy, even though they're likely on the same subnet. The same subnet as all of your iPads, MacBooks, and phones. Segmenting the network makes it harder for evil coming in on the, shall we say, vendor supported side from the more routine evils faced by general web-browsing.

Not that segmenting is easy to do, unfortunately.

InfluxDB queries, a guide

| No Comments

I've been playing with InfluxDB lately. One of the problems I'm facing is getting what I need out of it. Which means exploring the query language. The documentation needs some polishing in spots, so I may submit a PR to it once I get something worked up. But until then, enjoy some googlebait about how the SELECT syntax works, and what you can do with it.

Rule 1: Never, ever put a WHERE condition that involves 'value'. Value is not indexed. Doing so will cause table-scans, and for a database that can legitimately contain over a billion rows, that's bad. Don't do it.
Rule 2: No joins.

With that out of the way, have some progressively more complex queries to explain how the heck this all works!

Return a list of values.

Dump everything in a measurement, going back as far as you have data. You almost never want to do this

SELECT value FROM site_hits

The one exception to this rule, is if you're pulling out something like an event stream, where events are encoded as tags-values.

SELECT event_text, value FROM eventstream

Return a list of values from a measurement, with given tags.

One of the features of InfluxDB, is that you can tag values in a measurement. These function like extra fields in a database row, but you still can't join on them. The syntax for this should not be surprising.

SELECT value FROM site_hits WHERE webapp = 'api' AND environment = 'prod'

Return a list of values from a measurement, with given tags that match a regex.

Yes, you can use regexes in your WHERE clauses.

SELECT value FROM site_hits WHERE webapp =~ /^api_[a-z]*/ AND environment = 'prod'

That's cool and all, but the real power of InfluxDB comes with the aggregation functions and grouping. This is what allows you to learn what the max value was for a given measurement over the past 30 minutes, and other useful things. These yield time-series that can be turned into nice charts.

Return a list of values, grouped by application

This is the first example of GROUP BY, and isn't one you'll probably ever need to use. This will emit multiple time-series.

SELECT value FROM site_hits where webapp =~ /^api_[a-z]*/ AND environment = 'prod' GROUP BY webapp

Return a list of values, grouped by time into 10 minute buckets

When using time for a GROUP BY value, you must provide an aggregation function! This will add together all of the hits in the 10 minute bucket into a single value, returning a time-stream of 10 minute buckets of hits.

SELECT sum(value) FROM site_hits WHERE webapp =~ /^api_[a-z]*/ AND environment = 'prod' GROUP BY time(10m)

Return a list of values, grouped by both web-server and time into 10 minute buckets

This does the same thing as the previous, but will yield multiple time-series. Some graphing packages will helpfully chart multiple lines based on this single query. Handy, especially if servername changes on a daily basis as new nodes are added and removed.

SELECT sum(value) FROM site_hits WHERE webapp =~ /^api_[a-z]*/ AND environment = 'prod' GROUP BY time(10m), servername

Return a list of values, grouped by time into 10 minute buckets, for data receive in the last 24 hours.

This adds a time-based condition to the WHERE clause. To keep the line shorter, we're not going to group on servername.

SELECT sum(value) FROM site_hits WHERE webapp =~ /^api_[a-z]*/ AND environment = 'prod' AND time > now() - 24h GROUP BY time(10m)

There is one more trick InfluxDB can do, and this isn't documented very well. InfluxDB can partition data in a database into retention policies. There is a default retention policy on each database, and if you don't specify a retention-policy to query from, you are querying the default. All of the above examples are querying the default retention-policy.

By using continuous queries you can populate other retention policies with data from the default policy. Perhaps your default policy keeps data for 6 weeks at 10 second granularity, but you want to keep another policy for 1 minute granularity for six months, and another policy for 10 minute granularity for two years. These queries allow you to do that.

Querying data from a non-default retention policy is done like this:

Return 14 weeks of hits to API-type webapps, in 1 hour buckets

SELECT sum(value) FROM "6month".site_hits WHERE webapp =~ /api_[a-z]*/ AND environment = 'prod' AND time > now() - 14w GROUP BY time(1h)

The same could be done for "18month", if that policy was on the server.

The web of trust shrinks

| No Comments

Legislation is passing Congress right now, with a promise of a signature, to add new exceptions to the Visa Waver Program for a broader class of visitors to the US:

  • Nationals of Iran, Sudan, Iraq, or Syria, regardless of any other passports they hold.
  • Anyone who has traveled to those four countries within the last five years.

Because, terrorism.

Iran and Syria have expansive rules for who may be considered a national, which means people who have never been inside one of those countries may be governed by this new rule. Among those are Steve Jobs family.

The EU is promising Dark Vengeance (well, firmly worded disappointed words, followed by a possible reciprocal attack on US entry).

Because Canada and Mexico both allow US citizens a visa-free entry, most Americans have zero idea how travel to a visa-requiring country works. Or even what is required. It's specific to each country, there is sometimes an application fee, being denied entry does not get you your money back, and more and more countries are requiring biometric data (fingerprints, eye-scans) as part of the application or entry processes. It introduces friction to international commerce and travel, which is why the US introduced the Visa Waver Program in the first place.

But, terrorism.

And trusting your neighbors well enough to police their own borders.

And dealing with domestic cries to vilify whole peoples.

We are seeing the continued erosion of the US web of trust. The EU used to be a prime partner in just about everything; we spent so much rebuilding Europe in the 1940's and 50's, those relationships don't die easy. And yet, here we are, about to say:

We trust you to tell us who are bad people. But these people will require us to do the determining, sorry.

It's throwing sand into the wheels of commerce, a point the EU ambassadors have made.

That trust is a big thing. Participation in the Visa Waver Program is why EU passports have biometric chips in them now. In the background Travelers from waver countries still have their details run through the same electronic background check that visa-countries require. A country can't get entry to the waver program unless you meet some heavy requirements, some of which are political ("shared democratic worldview").

By forcing people from these four countries to go in person to a US Embassy and obtain a tourist visa for entry, we are greatly increasing the effort it takes to travel here. Business people in, for example, London will need to add at least 7 days to their travel prep-time in order to get an appointment at an Embassy; there will be no hopping on a plane with three days notice to go to a meeting in New York.

This is pandering to the domestic crowd at the cost of our economic flexibility, with no significant increase in our security.

Burocracy ~= Automation

| No Comments

That was one of the big points in this morning's keynote by Mickey Dickerson. Laws and the federal system is designed to be a big hunk of automation, executed by humans. Failures in the processes are entirely predictable, and like CPUs, the human processors likely can't do much about it.

So, so true.

And also pointed out that the US Federal System is a huge open-source project kicked off a couple hundred years ago with 538 comitters at the moment. There are lessons in big open-source project management in here somewhere, I'm sure.

It turns out one of the biggest goals of the US Digital Service is to ferret out the worst of the bad automation and try to turn it around. was their first big project, and right now they're making inroads with the Veterans Administration. They're doing some other things they can't talk about yet because government, but this is all worthy cause kinds of things.

During my last job-hunt I considered applying, but didn't go there due to where I wanted to live and an unclear answer on if they'd take a remote worker. Still, this is good stuff and I wish them all the luck and support.

Other Blogs

My Other Stuff

Monthly Archives