After several years working in on-premises data centers, I’ve recently returned to the cloud. This inspired my return to blogging, as much to play around with the newest AWS technologies as to write. I decided to build this site as “cloud native” as possible. This, by definition, excluded solutions such as Wordpress.

The goal is to not have servers to manage, as servers are single points of failure, require maintenance, security patching, general upkeep, and so on.

I’ve long been a reader of Daring Fireball and appreciate that Dan’s site is served as static content. This simplifies matters considerably, and I’m a big fan of simplicity. While Dan’s solution is built on Moveable Type and MT serves static content to visitors, the publishing toolchain is quite complicated to configure, install, and maintain. To make it cloud native on the configuration management side, I would need to write Terraform to set up infrastructure and then a considerable amount of ansible, puppet, salt, or chef to configure the server. That is exactly what I am trying not to do.

Terraform is a great solution because it allows us to define our infrastructure as code. This, in turn, allows us to simply change a config file and re-run the tool to make any changes necessary. For instance, if I chose a new domain name, I simply change the var.domain_name variable and re-run terraform apply et violà, my site is magically showing up at the new domain.

Jekyll is possibly the most popular open source static site generator at the moment, as it’s used for a larger fraction of the GitHub Pages sites out there. Its installation is relatively simple, being a single Ruby Gem. It uses Markdown for formatting, which is a huge bonus.

Infrastructure

The infra needed to run j.eremy is all Cloud Native, and hosted on Amazon S3. I used Terraform to configure the cloud components. The basic building block is an S3 bucket, which I configured as such:

data "aws_iam_policy_document" "website_policy" {
  statement {
    actions = [
      "s3:GetObject"
    ]
    principals {
      identifiers = ["*"]
      type        = "AWS"
    }
    resources = [
      "arn:aws:s3:::${var.host_name}.${var.domain_name}/*"
    ]
  }
}

resource "aws_s3_bucket" "website_bucket" {
  bucket = "${var.host_name}.${var.domain_name}"
  acl    = "public-read"
  policy = data.aws_iam_policy_document.website_policy.json
  website {
    index_document = "index.html"
    error_document = "index.html"
  }
}

By default, this makes a static website available at j.eremy.nl.s3-website-us-east-1.amazonaws.com. Of course, we’d rather it be available at our own domain name: j.eremy.nl. So we use Route53 to configure DNS. First we need to create the domain zone itself, and then the A record that points the j name to the S3 bucket.

resource "aws_route53_zone" "eremy_nl" {
  name = var.domain_name
}

resource "aws_route53_record" "j_site" {
  zone_id = aws_route53_zone.eremy_nl.zone_id
  name = "${var.host_name}.${var.domain_name}"
  type = "A"
  alias {
    name = aws_s3_bucket.website_bucket.website_domain
    zone_id = aws_s3_bucket.website_bucket.hosted_zone_id
    evaluate_target_health = false
  }
}

The only manual step, sadly, is updating my nameserver records at my registrar. In order to avoid spelunking through the AWS Route53 console to find these, I have Terraform output the list.

output "eremy_nl_nameservers" {
  value = aws_route53_zone.eremy_nl.name_servers
}

From this point, the files in the S3 bucket now appear at my domain name. It’s simply a matter of generating the site and uploading it to the bucket.

This doesn’t yet get us DOS protection or HTTPS, but we build things one step at a time in the Derr Household. All in good time.

Next, we’ll break down the publishing process.

I’m reintroducing my blog for a few reasons, and seeding it with the posts I wrote to commemorate the first anniversary of our arrival in the Netherlands. It’s now been three years, we’ve bought a house, lost a dog, I’ve become a goalie once again, I’ve changed jobs, and a pandemic has swept the globe.

The public transit here is very extensive and (generally) reliable. Being extensive brings with it a certain level of complication, and this sometimes mixes poorly with American senses of scale. There’s two metro stops and three bus stops within convenient (Dutch) walking distance from our house.

We didn’t have bikes for about our first week in Amstelveen, so we were reduced largely to walking and public transit, and we still hadn’t adjusted to the proper sense of scale of distances here. Looking at the map, we had the following options:

• 500m walk to Punter bus stop
• 1km walk to Meent metro station
• 2km walk to Middenhoven shopping center

In our heads, the shortest possible walk trumped all other concerns, so off to Punter we went.

I forget why we didn’t go to the Coop that’s a 500m walk from our house … but if I recall, it was closed on Sundays at the time, so wasn’t an option.

What we didn’t quite get is that the 174 bus only runs twice an hour. So we had to wait a little longer than expected for it … with screaming toddlers. And then the first bus was cancelled and we wound up waiting for nearly an hour at this cold, wet bus stop. Hungry children being hungry children, there was a lot of angst between both parents and kids. There was probably some yelling. But eventually, we got to the Amstelveen Stadshart, got some shopping done, and fed the kids (I think their first) McDonalds in the Netherlands.

We waited at the bus stop going the other way for a bit, only to see on the iPhone app that it was completely ambiguous whether a bus was ever going to show up or not … at all. So we found a random street to stand next to (near the library and modern art museum) and called an Uber. We picked the worst possible place to wait, and it took a couple of calls back and forth with the driver to actually find each other.

All while our children were tired and melting down… yet again. In the cold and rain.

So, a 1km walk felt insurmountable at the time - especially with toddlers and bad weather. A year later, while we try to avoid that walk … we totally do it all the time. It’s no big deal.

Now that we have bikes, a lot of this is immaterial: we can cycle (with kids!) up to a few miles quite easily to catch the appropriate metro, train, or bus, to get pretty much anywhere in the country quite easily. We also know the hours (“openingstijden”) of the local grocery stores each day by heart. That 5 minute walk to the Coop is a 2 minute bike ride - if we’re short on milk at dinner time, I can step out to get it and be back in 10 minutes.

Getting the 1km to Meent is just a few minutes on bike, as well, so we frequently cycle over there as a family, lock up, then take the metro (which runs at least every 15 minutes!). We can even bring our bikes on the Metro and use them for the “last km” at our destination.

For that first ill-fated trip, though, we’d have saved ourselves a lot of pain and tears and yelling if we’d just walked all the way to the metro. We’d thought about it - I think Kristin was even a bit mad at me that I thought the bus was the better option and just forged on ahead. A few days later we did just that, and it was absolutely terrible. The kids weren’t used to walking that far, we weren’t used to walking that far, we weren’t used to shepherding the kids for that long of a walk, and it felt like we’d just never, ever actually get there. But the more we did it, the shorter the walk felt, and now it’s just … normal. We do it all the time, if necessary.

Our primary mode of transport is bicycle. I mean… it’s the Netherlands, so that’s sort of a given. For trips around Amstelveen and Amsterdam Zuid, we’ll usually just go by bike entirely. For other trips, it just depends on how close the destination is to a train or bus stop; you can take your bike on trains and metros/subways, so sometimes we’ll bike to a metro stop, navigate metros and trains and bring our bikes along, and then use them for the “last” mile on the other end.

A few reflections on this.

Kristin’s Bike Woes

As you all well know, Kristin is absurdly short… and as you may not know, the Dutch are some of the tallest people in the world (#3!). So, bikes are built for tall people here. It’s hard enough to find an “adult” bike anywhere in the world that’s built for someone who’s 4’10”. Now, try to find that bike in the land of the tall people? Forget about it.

Kristin’s “sport” bike is made by Giant and is a size “XS” - and even then, she’s technically still an inch too short for it to be the “right” size for her. They don’t make an “XXS”, I’m sure because they’d sell exactly 4 a year. When you’re under the 5th percentile in heigh distribution, it really complicates things.

The first bike we bought Kristin was a cargo bike - a “bakfiets”. She rides it to this day. Just getting it home was a 3 mile ride and nearly killed Kristin - even at the lowest setting, she could barely reach the peddles. Now, try riding a giant, steel cargo bike, loaded with 2 children, when you can barely reach the peddles?

There were tears shed that night.

We learned eventually that you can reverse the mounting bracket on most saddles and earn yourself about one more inch. This helped, but to be fair it just took the bike from being “impossible” to peddle to just “somewhat uncomfortable”.

We’ve visited bike shops (“fietsen”) from one end of the country to the other looking for bikes that Kristin can safely ride with a child seat on it - no mean feat. Most women of her stature simply ride a girl’s bike, but those often aren’t suitable for one child’s seat, let alone two. Ultimately, we found a German bike at a fietsen in The Hague that, like her Giant, is “technically” too big for her, but she can safely and comfortably ride with a child on it.

We ordered it 2 months ago and right now it’s still about a month out from delivery. So, Kristin continues to use the bakfiets around Amstelveen but when we go on longer trips I take both children on my bike and Kristin rides her sports bike.

Flat Tire #1

The first flat tire was suffered on Benjamin’s first day of school. She cycled the children to school - a 2.5 mile trip - in the bakfiets, only to have a front tire go flat about 200 feet from the school. We’d been in the country for about a month and a half at this point, had never tried to repair a bakfiets, had no idea where a fietsen was in that neighbourhood, and hadn’t really figured out what the public transit options were in that neighbourhood. Consequently, tears were shed.

In an emergency, in the US there are tire fillers you can get that fill your tire with foam that will at least allow you to complete your current trip. I set about trying to find some of this, only to discover it’s just not a thing here. At all. I even went to a fietsen and asked in person, and they looked at me like I had grown an extra head. He handed me an inner tube patch kit.

Patching an inner tube is a lot of work. You have to remove the wheel, then use a pry tool to remove the tire, then inspect the inner tube to find the hole (often using soapy water, then looking for bubbles as you put air in the tube), patch, wait, reassemble. But here’s the thing I didn’t really understand at the time: very few dutch bikes use derailleurs and instead have internal hub planetary gears. These are almost exactly zero maintenance and last decades … but you DO NOT EVER take your back wheel off in the field. Ever ever ever. Removing these wheels is the stuff of seasoned technicians. As a result, the Dutch are pros at patching tubes without even removing the wheel. You pop the tire off, then just listen for a whistle while you pump air into it. The Dutch will patch a tube endlessly, never replacing it until it’s more patch than tube or dry rotted beyond its useful lifespan.

For this episode, I knew none of this, so Kristin locked it up near the school and I retrieved it after work and walked it about half a mile to a fietsen who fixed the tire for about €25.

Kristin’s rear tire went flat at one point and I tried removing the planetary gear set. To say this was a disaster is an understatement. I didn’t even vaguely have the right tools for the job, and we wound up having to have the manufacturer send a technician to our house to fix my error. I forget the exact bill, but I think this cost us about €200 parts and labor.

How I learned to patch tires in the field

For a while, I rode a gold 60s or 70s “Dutch special”. This bike powered me through the spring and carried me and both kids through our summer adventures. It was a workhorse, but has since been retired in favour of my much more comfortable tourer from German manufacturer VSF.

It also had internal gears, and I acquired it with tires already warn to the radials. Gun-shy from my previous experience with Kristin’s bike, I very lazily refused to deal with this situation as long as humanly possible. This went on long enough that Kristin was actively hassling me to get my tires replaced.

Well, the expected outcome came to pass. Through the spring and early summer, I would cycle all the way to work - about 8 miles! And one day, I hit a rock right on the baldest spot on the front tire, puncturing the inner tube. I was passing Station Zuid, about half-way home. We owned a patch kit - but I did not have it with me, and I didn’t have time to walk 10-15 minutes each way to the nearest fietsen.

I locked up my bike, hopped on the metro line (thankfully about 20 feet away) and headed home. After getting the family through dinner, I headed back out. I popped the front tire off, removed the tire and patched the tube. In under 10 minutes, I was back on my way. Now, if I’d been smart, I’d have taken my bike on the metro for the bulk of the trip home. But I’ve never been one to learn lessons the easy way.

About 15 minutes later, just short of the Amstelveen Stadshart - about half way from Zuid to home - I hit another obstacle on the fietspad and punctured the tube … again! I flip my bike over, and in 10 minutes I have it re-patched, and I’m on my way once again.

I don’t get maybe another 200 meters, just on the other side of the Stadshart, when it punctures yet again. I still have another 10-15 minutes to go to get home under the best conditions, and decide to throw in the towel. I lock my bike back up at the Stadshart, then take the Metro home yet again. On the ride, I order a new set of tires for overnight delivery. The next evening, I run back up to the Stadshart and replace the tire and inner tube, and get home safely.

I still haven’t replaced the rear tire on that bike, which is sitting in front of the house to this day.

The Time My Bike Got Stolen While Buying Some Hockey Gear

Immediately after we rented our house out in Amstelveen and moved out here, we realised that we needed at least one bike: walking to/from the local shopping centres was just far enough away that walking in the winter is a pain in the ass. On a blustery Saturday morning, I walked about 2 miles to the nearest fietsen I could find that I knew would be open and way overspent on a brand new city bike. It was a good bike that lasted me about 2 months? Maybe 3. I forget exactly.

As we were finally getting into a routine, I decided to re-gear myself for hockey and found a guy from the local 4th division team who was hanging up his skates for good. He happened to live very close to us - about a 10 minute bike ride. He sold me all of his gear for about €100.

I thought I was going to knock on his door, hand over €100, and get handed a bag of gear, so I was lackadaisical in bike security. Between the moment I rang his doorbell and when he opened the door, a torrential rain started. It went from “it might rain soon” to “THE END TIMES ARE UPON US” level rain in the blink of an eye.

So, when he opened the door, he quickly invited me into his entry to perform our transaction. In my panic, I left my keys in the chassis lock.

And in the 5 minutes I was inside, that hockey gear wound up costing me €100 in cash and a €600 bicycle.

Getting your drivers license is more involved here. I can skip most of the bureaucracy due to my visa status, but I still need a valid health certificate.

The questionnaire is not available in English, and both I and Google misunderstood a verb tense. The question was “Are you being treated for ADD/ADHD?” but I thought it was “Have you ever been….” and answered Ja.

If you answer Ja to anything, you have to get a doctor to certify that you’re not a risk on the roadways and are being appropriately treated by a medical professional.

Oops.

I didn’t have a GP and certainly didn’t know how to get an appointment with a psychiatrist. I let it go, as simply too much trouble given I wasn’t expecting to drive … maybe ever. But the DMV eventually sent me a referral - in English, no less.

I went to see this doctor and it was sketchy as hell. He was in an office that could have been assembled in 3 hours and bore a passing resemblance to the set of a well known porn production involving casting couches. Cash only.

He also fit the stereotype of the Greek or Turkish mobster, and despite wearing a well tailored suit, had egregious amounts of body hair coming out of … everywhere.

The interview went on as you might expect an ADHD evaluation to go. Despite the above, very professional. Until the end.

“So, uh… hyperactive uh? You have problems with impulse control?”

“Uuuuuuh…”

“You know. Doing things. Not like … the impulses of man.”

With that last he made kind of an “uungh” sound that was very uncomfortably sexual in tone.

“Uuuuuuuh,” I repeated, and then my impulse control failed me. “Does it count if I channel my poor impulse control into the impulses of a man?”

The silence stretched forever and ever. He just stared at me blankly. I knew, before I’d finished what I was saying, that I’d said the wrong thing.

But then … “HAHA! I like you! You funny! Ha! You’re good! You drive! Have a good day!”

From the archives, I bring you a story from our earliest days in the Netherlands. In fact, our very first few minutes.

One of the constant themes of our immigration to the Netherlands has been “baby steps.” Sometimes, learning how to do something new “The Dutch Way” (or simply something we’ve never had to do before) feels insurmountable because there are so many unknowns along the way. So we have to step back and solve them one-baby-step-at-a-time.

The mere act of arriving at Schiphol Airport was perhaps the greatest example of this.

There are a multitude of stories that precede this one: the travel from Chriesman to IAH was not without crisis. Just getting from the rental van into the airport was nearly ruinous. Penny has been sick the entire trip thus far, but mercifully slept between her bouts of puking. Ben, on the other hand, refused to sleep until the last hour and a half of a 10 hour flight. Upon arrival, we’re already at the end of our ropes and neither Kristin nor I has slept in almost 24 hours.

Passport control was, thankfully, a breeze. The immigration officer asked a few basic questions, welcomed us to our new home, and waved us through. We then had to retrieve our luggage (two VERY over-stacked dollies full) and our two very-large dog crates (plus, of course, dogs). The two customs officers we worked with to “clear” the dogs looked at each other like they both knew we were asking them to do something against the rules, but they helpfully opened a door that was “closed” and helped us wheel our belongings and animals out of the customs area… but they literally stopped like a foot outside the door. They pushed out carts out the exact distance required to close the doors again behind us and that was it.

Now… You have a very cranky 3 year old who’s operating on an hour and a half of sleep and a sick 18mo daughter who is obviously not in the best mood. Two sleep deprived adults. And four luggage dollies that are stacked to the point of turning over. And you have to get all of this from one end of the airport to the other, and ascend to the next level along the way.

For an absurdly long period of time, we thought we could use the River Crossing Puzzle to logic our way to a solution.

I’ll take the kids up to the taxi stand, leave them, then … uh … wait, can’t leave the kids. I’ll take the dogs… No, that won’t work either. I’ll take the lugga… shit, what if someone steals it? Wait, first Kristin takes the cabbage and comes back, then I take the wolf… Dammit, now we’re not even sure what we’re talking about.

“Look, we’re supposed to check in at this limousine desk near the taxi stand. They can find us help. Somehow.”

Kristin wasn’t buying it, and honestly I don’t blame her. After quite a bit of bickering back and forth in our sleep deprived state, we simply cannot find another solution. I won’t lie, there was probably some yelling, and half the Netherlands probably still remembers us as that wicked-pissed-American-couple who didn’t have their shit together. But we soldiered on.

We moved ourselves into a little corner, out of everyone’s way, and then I abandoned my family and everything I owned and loved in a strange airport and set off for help.

One of the first times I visited Kristin’s apartment when we first met, she went to the restroom and I made it my mission to catch her elusive anti-social cat Cookie while she was gone. When she came back, there I was on the couch, holding Cookie, and neither I nor the cat looked even slightly at ease. In fact, I’m pretty sure I looked like I knew… if I loosened my grip, this cat was going to skin me alive.

That’s exactly what Kristin looked like when I walked away from them, setting out across the marble plains, my own private Schiphol Trail, in search of help. Hopefully no one died of dysentery or got swept away attempting to ford a river.

My logic was this: it was the JOB of the people at this booth to help newly arrived immigrants get their asses out of the airport. If anyone in this entire airport knew how to get me, my wife, two ornery children, two bewildered yet crated dogs, and two carts full of luggage from Customs to Taxi stand… it’d be them, right?

Well, not so much. He just sort of looked at me, and I had to explain it again, in smaller English. That’s not a dig on him or his command of English, but more about my addled state at this point. While he did not have a ready-made solution - there weren’t any porters or anything to speak of, not on this scale apparently… but he did have a gung ho attitude and was immensely helpful. Turns out the guy at the window was just a dispatcher. He waits for people to show up, and calls them a taxi, van, or limo. That’s his job.

Well, he went above and beyond today. He personally came out of the booth, followed me back across the entire airport to my family, and took two whole carts to himself. He even knew a shortcut to get from where we were to where we were going. I forget how Kristin and I divided things, but I think we each took a cart and child, and forged ahead with our pioneering dispatcher in the lead. A scout of unimaginable skill.

One last challenge awaited us: they had called us a large Mercedes Benz shuttle van to get us from the airport to our temporary apartment, but the dog crates were far too big to fit into the van along with our luggage. But you see, our enterprising guide immediately fell back upon his actual job: within just a few moments, he dispatched a second van for us. I took dogs in one van, Kristin took children and luggage in the other, and we headed off to do our best to get over our jet leg…

Genius Bar: The Apple Store’s Fading Legacy?

Let’s rewind to the summer of 2002. Apple’s retail initiative has just passed its first anniversary and, in September, opens its 50th store in the US. The original iPod is mere months old and, largely, is a curiosity still looking for the insane success it would soon become. Apple has just shipped Mac OS X 10.2 “Jaguar” and the new hotness is the iMac G4.

Genesis of the Genius Bar

The Genius Bar was originally conceived as a one-stop shop for customers to get training, troubleshooting, and repair for their Macs. At just fifty Apple Stores, Apple only needed the four or five hundred Geniuses across the country and was willing to splurge a little on salaries to make for a stellar experience. Many of the first two generations of Mac Geniuses came from inside Apple, including myself. This created a pool of employees who were already instilled with the levels of service and caring that Apple demands.

Early on, working the Genius Bar was very nearly a sacred duty. We all felt the weight of the customer experience. We knew that every word we uttered, every action we took could have a tremendous effect on whether this customer would remain a loyal Apple customer. We spent considerable time evaluating, re-evaluating, and re-re-evaluating the tiniest of tiny details that would help or hinder interactions. Stores discussed best practices amongst each other, quickly getting new ideas into varied settings and deciding when and where to employ various tactics, arrangements, and logistics. There was but one focus: Customer Satisfaction. Nothing else mattered. At all. No schemes, no run-arounds, no qualms about exceptions. While other divisions of Apple had stringent requirements for replacing computers, our directive was far more lenient.

How The iPod Changed the Genius Bar…

The shape of today’s Genius Bar has more to do with the iPod than anything else. Whereas a Mac Genius in 2002 spent most of his time, logically, working with Macintoshes, the Mac Genius of 2006 spends far more time staring blankly at iPods. The 2002 Model Genius spent quality time with two, maybe three customers per hour. In 2006, he crams six or eight iPods into an hour and commonly spends an entire day handling only shiny white music players while a co-worker helps the Macintosh customers. In 2002, the Genius helped mostly level-headed, professional adults. Later, he helps mostly tantrum-throwing children and teenagers.

iPods are decidedly simple devices, and no one within Apple does component-level repair on the darned things. Whether you mail it in to Apple for repair or you take it to Apple, you get a new iPod in return. Geniuses aren’t in the back room tearing iPods apart and replacing screens and drives. The handling of iPods is completely and utterly opposed to the handling of Macs. When assisting a Macintosh customer, the Genius uses an often complicated decision tree to reach a solution. Do you just need training, or does your Mac need some work done? If it needs work, will the problem require leaving the computer at the store for a few days? If it does need to stay, the Genius must thoroughly document all steps large and small, as he will likely not be the Genius actually performing the repair.

For iPod customers, the possible outcomes are much less diverse. If it’s broken, he trots backstage and gets a new one for you. If it’s not, he doesn’t. Most of the interaction becomes paperwork, not training, troubleshooting, or repair.

…and How It Killed It

Apple hired a metric boatload of Mac Geniuses for their (say it with me now) training, troubleshooting, or repair. They hired imminently qualified and capable individuals to do this, and they paid them handsomely. A while later, this great new product came along and suddenly they were doing little more than pushing paperwork.

Well-paid, well-trained individuals don’t often handle reductions in duties very well. They want to grow intellectually. They want to grow professionally. They want to show their skill to their co-workers and customers. They want to be heroes. Instead, they’re processing paperwork. Can you guess what their reaction is? I can.

They leave. In droves. Among the many negatives that cropped up, they all have different camel-breaking straws. Some are refused long-promised promotions, others have critical blow-outs with customers, and yet others simply burn out. But the whole story is that they were hired for a fairly important job, one that they performed well for nearly two years, and then were asked to stoop to paper-pushing and baby-sitting.

The experience left and, sadly, was not replaced by equal talent. As the well-paid Geniuses left, Apple Retail took the opportunity to save on payroll by hiring replacement Geniuses at deeply discounted rates. Watch out for falling prices, as they say, and that meant watching out for falling satisfaction levels.

Can It Be Fixed?

Apple faces important questions, chief of which is whether they can fix the Genius Bar. The Bars have become magnets of negativity both for the customers and for the Geniuses themselves, largely because of tremendously increased waiting times and decreased quality time with the Genius. Treating the Genius Bar as a cattle mill of customers negatively impacts the Geniuses, as the phenomenal number of faces they help become impersonal and uncaring. Waiting longer leads to customers being more curt when their turn arrives. When a Mac Genius is processing iPod Warranty claims, he’s ignoring a part of his job that is so important that it’s in his title.

The Mac.

Maybe Apple will remember this, some day. Until that day, the Genius Bar will continue to suffer. As it stands, the Geniuses spend an inordinate amount of time dealing with iPods, a task that doesn’t really need the attention of a Genius. Anyone can plug in a serial number, enter a note, and hand the customer a new iPod. Not just anyone can help a customer set up WDS or troubleshoot audio sync issues in Final Cut.

Apple should return the Mac Genius’s focus to the Mac, where it belongs, and it should start paying the Geniuses rates that correspond to that renewed focus, brining in new Geniuses with significant work experience. They should hire “iPod Geniuses” to handle iPods, or institute a different support structure altogether for iPods. Regardless of the actual mechanism, the focus of the Mac Genius must be, as their name explicitly states, the Mac.