I’m spending an hour or more running on most days now, with a couple hours and steadily increasing each Saturday. I’m wanting to get back to spending time in nature — backpacking, hiking, and other activities that take us away from civilization. The issue with that, is that I’m always on call. Kelly and I make our living two ways. First, I have a work from home position as a Network Security Engineer for a large corporation, that provides our base salary. It’s flexible, but demands 40+ hours a week of my attention just like any other full-time job. I love the company and the team I work for – there have been some rough patches, but the only way I’ll give up this position is if it’s forcefully taken from me. Second, we have our own consulting and hosting company. Between the two, we have enough income to live comfortably — but if we lost one or the other, we’d eventually have to find a replacement. For my day job, a coworker and I have arranged a rotating series of on-call shifts. The on-call load is relatively light, and it’s usually pretty easy for me to arrange to not be on call if I’m going to be busy, by just giving him a heads up. He tends to stay close to home and works in the UK, where he gets paid extra for being on call, so he is usually happy to soak up these hours. On the flip side, for the consulting company, all the on-call work is on me. I have a friend who will jump in and take care of things from time to time, but he’s not a full time employee – rather an occasional contractor – and I can’t pay him to sit around waiting on things to break. Not to mention, he has other things that demand his time. We have two racks of equipment for our hosting services. The majority of that equipment is several years old – architected on a strict budget, with many ad-hoc changes made over the years, and too many single points of failure for me to feel comfortable taking an extended leave of absence. We have some customers who rely on our hosted services for their daily workflow and income, pay us accordingly, and demand continuous uptime. We moved into the first rack back into 2011, and the contract is coming up for renewal again this June. Since we’re negotiating a new contract, or a possible move to a new facility, I decided to take a fresh look at our architecture to alleviate all single points of failure. My new goal is to make it so that if failures occur, I don’t have to be around to immediately address them. There are a few things in the racks that belong to customers – we don’t necessarily take responsibility for them, and if the hardware has issues, then those issues are theirs to repair. I’ll disregard those in this design, if I can’t convince the customers to move them on-premesis or to move to high-availability solutions. First is addressing the switching infrastructure. We currently have a single switch in each rack. I’ve spent the last few years working with HPE Comware equipment for my primary employer, and have been really impressed with the hardware. Because we have a mix of 1G and 10G equipment, I ended up buying two 5900AF-48G-10XG-4QSFP+ and two 5820AF-24XG switches. That’s moving from one 48-port switch per rack, to four switches (hoping to condense both racks to a single one, to reduce monthly expenditures). These HPE switches support IRF, Intelligent Resilient Framework, a type of stacking that allows for advanced configurations such as LACP bonding across chassis. The switches themselves have redundant fans and power, and if in pairs, you can connect servers to both switches with bonding, to survive complete switch failure, cable failure, optic failure, and a few other usually catastrophic scenarios. I have one IRF stack containing the 1G switches, another consisting of the 10G switches, and a 4x10G bond between them. These switches also support BGP and advanced hardware ACL processing, which allows me to alleviate the redundant routers we’re currently running. I’ll be replacing the services the switches can’t do with virtual routers. All of the described infrastructure below will be exclusively 10G, except for management and out of band. Next, I wanted to replace the aged VMware infrastructure that consists of several individual servers with some cross-server replication, with a true HA cluster. I ended up ordering three servers that contain dual 8-core CPUs, 384GB RAM, and 4x 10Gb NICs each to start with. They’re running Proxmox, which has been configured for high availability. Networking consists of two bonds that stagger between both NICs (each bond consists of a port from each of the two cards) and switches. The first bond is for data, and the second dedicated solely to storage. This arrangement gives me N-1 RAM capacity, or 768GB to start with, to maintain full ability to sustain a node failure. I currently have about 500GB in mind to migrate, so it will give me a moderate amount of room to grow. One of the great things about this solution is that I can add additional hosts at any time – and even re-task some of the existing hardware in the rack to serve as hosts in this cluster once I have the guests migrated to the new infrastructure. Doing so would be a free or low-cost way to add another half terabyte of RAM, or more. Last, and the thing I am most excited about, is the storage solution. I would have rebuilt the hypervisor solution a long time ago if I had a centralized commodity storage solution I was comfortable with, or the ability to invest in an off the shelf one that would meet all my requirements (they start in the tens of thousands and skyrocket from there). Ultimately, was able to engineer a solution that checks off all the boxes, using technologies I’m already familiar with, by accepting that a couple key technologies I rejected a long time ago have since become viable. The end result physically consists of two storage heads running Debian and dual-controller SAS expanders (which we can easily add at any time to increase capacity) that hold the drives. Without getting in a ton of detail that is beyond the scope of this blog, these are a few of the failure scenarios I’ve simulated while running intensive stress tests and/or benchmarks, without issue: Cutting power to each 5900AF-48G-4XG-2QSFP+ switch (one at a time) Cutting power to each 5820AF-24XG switch (one at a time) Bulk live migration of VMs Cutting power to a host with active VMs (they die, but automatically boot up on other hosts) Cutting power to the active storage head Cutting power to the standby storage head Pulling the active SAS controller out of the expander Pulling the standby SAS controller out of the expander Randomly disconnecting network cables for A or B side connections Cutting A or B side PDUs For all these solutions – switches, hypervisors, and storage – another upside is that we can perform upgrades and maintenance without impacting customers. I’ve spent a week building this – the last three or four days of which consisted of extensive load/stress testing, and failure simulation. I have tested several failure scenarios, under heavy load, with excellent results. The new stack will hopefully be racked in the next month, if early contract negotiations with our datacenter pan out as I hope. Once this solution is in place, VMs migrated over, and the old gear retired, I’ll be a lot more comfortable being out of reach and away from the computer. I’m sure at least a few of our customers will be happier being on true HA solutions. Performance looks like it will increase nicely. It’s a win-win, really, even though it did cost a pretty penny in new hardware. I’ve been avoiding this for a long time – not wanting to invest, not believing that the technologies needed for the storage solution were mature enough (ZFS on Linux, primarily) – but technology is always changing, as are customer requirements, as are our own needs and wants. This is going to be an awesome upgrade both in technology, and though it might seem unrelated to the uninformed, quality of life for the two of us.
Daily Breakfast
I don’t really consider myself a breakfast person. A lot of people tout it as the most important meal of the day, but historically it is one that I’ve skipped far more than I’ve consumed. Since starting to be more active, I’ve started making smoothies when I wake up, unless I’m substituting pancakes or another special dish in their place. My recipe is more or less a basic guideline, based loosely on a No Meat Athlete blog post, that I sub ingredients into for variety. The basic outline, which serves two people, is: 3 frozen bananas (we buy them fresh, allow them to ripen, peel, quarter, and vacuum seal them three per bag) 2 cups of unsweetened frozen fruit (usually two varieties) 2 tbsp Flax Seed 1/4 cup fresh walnuts OR 4 scoops Orgain Vanilla Bean Protein Powder 2 or 3 large handfuls of spinach My favorite combination is peach, pineapple, and Orgain. Other fruit we keep in the house and mix-n-match are blueberries, blackberries, raspberries, strawberries, rhubarb, mango, cherries, and oranges. We usually reserve the Orgain for use on Saturday, which is our long workout day. One final thing to note is that several years ago we went through several cheap blenders making smoothies, dips, and creams, burning up the motors or just not being satisfied with the results. Eventually we invested in a Vitamix 5200. That was seven years ago – and it still works flawlessly. We recently upgraded to a refurbished 5300 with the newer pitcher design, and delegated the 5200 to the RV. I highly recommend investing in a Vitamix – they are amazing machines and are built to last a lifetime of heavy use.
Foam Rolling & Roll Recovery
We haven’t had much to post about lately, as we’ve been getting through the holidays and getting back into the groove of normal life. Things have been both dull and busy – a lot of monotonous catching up to do, mainly, with a few short memories sprinkled in, like spending new years with some great friends of ours from out of town. Kelly’s ankle is healing up nicely and she’s back to riding regularly and starting to jog again, and I’m still working to tick off at least 24 miles per week. My running coach has been after me to start foam rolling and focusing on recovery for a while. We have all the gear required for foam rolling, which consists of a few foam cylinders in different sizes and some lacrosse balls. Despite my best attempts, I can’t seem to make a routine of it. You have to get on the floor, or maybe a mat or blanket, and roll around while supporting yourself and manipulating your body to put pressure on sore areas – something that can be a substantial and time consuming, not to mention painful, workout itself. Roll Recovery makes an alternative solution that accomplishes the same thing. Touted as a “self-massage tool [that] takes the extra effort out of an intense foam rolling session“, it basically is a vice that you clamp your limbs into move along the length of them. It doesn’t address the last complaint I made about foam rolling – the pain, possibly even being worse – but does a great job of reducing the effort and time required. The R8 seems to have a cult following and a lot of obsessed users, so I figured I’d try it for myself. So far, I’ve found I pick it up when idle throughout the day and use it while watching TV or even listening to phone calls (meetings). That sure beats blocking out a 45min window to toss around on the floor while being able to concentrate on nothing else. Kelly is making similar habits with it. We also picked up the R3 and their stretch mat. The R3 is a foot roller, specifically designed to address plantar fasciitis, something I’ve recently began to experience, and the mat is a gimmicky number that gives you a platform you can stretch on without falling off of – something that may work for Kelly, but doesn’t seem to work for my 6’0″ frame. Despite that it may be a tad too small for me, and that I don’t mind sitting on the ground most of the time, it seems perfect for throwing in the truck and taking to the lake or trailhead with me. To date, as you probably assume, I haven’t been great about massage or rolling or recovery. I don’t have enough previous recovery experience to compare with, but what I can say is that my run yesterday (after two days of using the R8) was one of the best ones I’ve experienced in a while.
Sample Photos
We took the new camera out and took some test photos today. It’s a pretty complicated piece of equipment, so we have a lot of learning and improvement ahead of us. We hoped to see some birds, or something we could use the 70-300mm lens for, but nothing interesting came up.
Photography Gear
We’re trying to transition from collecting things to collecting memories – and capturing those memories is something we prioritize. Here is our new equipment to capture moments. … Continue readingPhotography Gear
Tyler State Park
Both of us have family in the Tyler area, so we go back to visit from time to time. A couple years ago we stayed at Tyler State Park, but have avoided going back there because it was really tight to get our rig in and out of the campground. When recently looking at the map, I realized that was due to the camping area we stayed in – Cedar Point. So when it came time to book a trip to East Texas for Thanksgiving, we gave Tyler State Park another chance and reserved a site in the Big Pine area. The park itself is beautiful woodlands, and this year was exceptional – the trees turned bright colors like they do up north, and leaves fell to the ground. The camping area we chose had plenty of room, all pull-through sites with full hookups, and ran $24/night. We consider that a good deal. What led me to looking at the Tyler State Park map in the first place was an ultramarathon called Running the Rose. I’m not ready for an Ultramarathon yet – in my training, I might be able to complete – but not be competitive – a half marathon at the moment. But Running the Rose follows an eleven mile course through the park, allowing you to exit at one, two, three, or six loops depending on your fitness level and dedication. I wanted to run the course, once, just to see if I could make it. I mapped out the course in GAIA GPS, and uploaded it to my watch. The course looks like this: Upon arriving, we found our spot and set up camp. Since we arrived on a Sunday afternoon, we had most of the park to ourselves. The site was spacious, though not at all level, and had full hookups. Fortunately we just about have setup down to a T, and I was able to head out for my first trail attempt within an hour of arrival. I hit the trail around 4:30PM, and shortly after five it was so dark under the tree cover that I couldn’t see the ground. I pushed through about 8 miles before my headlamp died, and after tripping over a bunch of tree roots, pulled out my phone for enough light to get back to the campsite. I clocked just over 9 miles, and kicked myself for getting so close to my goal and not reaching it. We had a friendly little visitor come that night, as well as a opossum and some squirrels. We spent the next few days hiking the park and visiting family. We grabbed a few good photos, and had my parents out for a campfire. Kelly brought out her bike for a few rides. The weather stayed really nice. On Thursday, I made another attempt at the 11mi run. I was able to complete it, but it took longer than I had hoped, clocking in at 2h32m and an average pace of 13:45/mi. I didn’t really feel up to running that day, and think if I were feeling better I could have knocked a couple minutes off that time. C’est la vie. On Friday, Kelly and I went on a run on the trail around the lake and she caught her foot under a tree root and pulled. It caused her a pretty nasty sprain that will take a few weeks to recover from. We got her back to the site and settled in, then I resumed my run. Overall, I was able to clock 32.5 trail running miles on the trip, plus several miles of hiking. Tyler state park is definitely a place we’ll be returning to. I’m contemplating an actual attempt at Running the Rose, when it takes place on January 26, but due to how soon that is, probably won’t commit to it. We have family there, so there are always reasons to go – but there are other places in Texas we want to check out before the next time to visit.
Fresh Start
Since starting this blog back in May 2016, we’ve accumulated some 206 posts. That comes to average one post every 4.5 days, which seems impossible – we must have gone through a few periods of heavy posting to offset the periods I remember more clearly… the periods of not posting at all. Over those two and a half years, which were kicked off by the purchase of our new fifth wheel RV, and the beginning of the process to modify our tow vehicle to meet our expectations, a lot has changed. We briefly got into crypto, made bit of money, and cashed out just in time to beat the market crash. We bought and restored a boat, only to find we’re not comfortable using it (too large to launch with one person, too unwieldy for small, usually low, lakes we like to visit) and are contemplating selling it to cash out and possibly downsize. We went through numerous phases that eventually faded to an end, such as trying to turn our suburban lot into a sort of homestead (not practical with HOA regulations and less than a quarter acre), fairly extensive DIY home renovations and updates (enough is enough), ham radio (still using VHF, but have sold our HF gear because we gave up on antennas here), and many more. In the last six months or so, we’ve discovered new fitness activities, and they’ve moved to the forefront of our lives. Kelly has really fallen in love with endurance cycling, which is big change from the years that she’s spent embracing and evangelizing ToneItUp. I’ve gotten off the couch and found endurance trail running and aspire to at least cross the finish line of a fifty mile or longer ultramarathon one day. We cross train in each other’s sports to spend time together and better grasp our own. My health has improved to the point where I feel like spending extensive time outside the house – doing things I love, like backpacking – are on the table again. These things take focus, dedication, time, and yes – changes to almost every aspect of your every day life. Going back through old posts, we saw a lot of fruitless endeavors, and very few posts about the things we created this blog for. We wanted to document our RV adventures, to record where we went, what we thought, how we’d change things and spend time differently; what sites we might request — if we visit those places again. We planned to take and post lots of photos (something we never do – we have DSLRs and drones we always plan to use, but we fall back to easy-access cell phones all too often if we remember to take photos at all) of the nature we witness and how places vary as you travel throughout the country. That is why we called this LifeEnRoute – it’s to document our life traveling in the RV, not the life of gadgets and gizmos and gear that really seem to fill all the posts we’ve made. So moving forward, we decided to archive them. They’re hidden, but still available for us to pull occasional content from if we so choose. We are, at some level, gear heads – and I mean that in both the conventional ways that phrase is used. We love all sorts of vehicles, all sorts of technology, all sorts of modifications — pretty obvious if you look at my truck –, and we love a lot of the latest and greatest gear and technology — pretty obvious if you look at our running and cycling gear (for example). But we’re chipping away at the contents of our closets and garage and finding the things that don’t matter so much now that we’ve had them, and unloading them to new homes via Craigslist, Offerup, and Ebay. Some time in 2019, we may do something a bit drastic. We may sell our house, store our furniture, unload our clutter, and embark on the journey we dreamed about back in May 2016. In our early 30s, and able to work 100% remotely, we have the rare opportunity to tour the country and take in all it has to offer. It’s something that a lot of people aspire to do, but most have to wait to retire to experience. With our new fitness aspirations, came new motivations and goals – as a trail runner, for instance, I know of tons of trails I would love to run and places I would like to train, that are scattered all over the country. We can target our travels to tick them off, one by one. It’s not a permanent lifestyle for us, but it seems like we can fit it between our starter home and forever home, which is a jump we will inevitably make. 1350 square foot and a a fifth of an acre aren’t what we’re looking for anymore. And we think if we do it, despite our few remaining reservations about it, we’ll find it to be great.